trentmkelly/LessWrong-43k huggingface dataset:
Hard problem? Hack away at the edges.
by lukeprog — 2011-09-26T10:03:41.197Z

Wei Dai offered 7 tips on how to answer really hard questions:

Don't stop at the first good answer.

Explore multiple approaches simultaneously.

Trust your intuitions, but don't waste too much time arguing for them.

Go meta.

Dissolve the question.

Sleep on it.

Be ready to recognize a good answer when you see it. (This may require actually changing your mind.)

Some others from the audience include:

Transform the problem into a different domain.

Ask people who have worked on similar problems.

Decompose the problem into subproblems. (Analysis)

I'd like to offer one more technique for tackling hard questions: Hack away at the edges.

General history books compress time so much that they often give the impression that major intellectual breakthroughs result from sudden strokes of insight. But when you read a history of just one breakthrough, you realize how much "chance favors the prepared mind." You realize how much of the stage had been set by others, by previous advances, by previous mistakes, by a soup of ideas crowding in around the central insight made later.

It's this picture of the history of mathematics and science that makes me feel quite comfortable working on hard problems by hacking away at their edges.

I don't know how to build Friendly AI. Truth be told, I doubt humanity will figure it out before going extinct. The whole idea might be impossible or confused. But I'll tell you this: I doubt the problem will be solved by getting smart people to sit in silence and think real hard about decision theory and metaethics. If the problem can be solved, it will be solved by dozens or hundreds of people hacking away at the tractable edges of Friendly AI subproblems, drawing novel connections, inching toward new insights, drawing from others' knowledge and intuitions, and doing lots of tedious, boring work.

Here's what happened when I encountered the problem of Friendly AI and decided I should for the time being do research on the problem rather than, say, trying to start a few businesses and donate money. I realized that I didn't see a clear path toward solving the problem, but I did see tons of apparently relevant research that could be done around the edges of the problem, especially with regard to friendliness content (because metaethics is my background). Snippets of my thinking process look like this:

Friendliness content is ab...
[SEQ RERUN] Original Seeing
by MinibearRex — 2011-09-26T02:46:09.669Z

Today's post, Original Seeing was originally published on 14 October 2007. A summary (taken from the LW wiki):

One way to fight cached patterns of thought is to focus on precise concepts.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was The "Outside the Box" Box, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
LessWrong gaming community
by TwistingFingers — 2011-09-26T02:19:20.734Z

Many of us enjoy expressing ourselves through electronic games. As such, I feel that this aspect of our lives should be shared among our fellow gamers in the LessWrong community.

Video games are a great way to reduce compartmentalization and learn real-world rationality skills. Indeed, what brings us together at LessWrong can often be our love of games; someone in the LessWrong community without this advantage might find learning rationality difficult. In this light, outreach into the transhumanist/rationalist community to promote gaming is low-hanging fruit for serving the future of humanity.

Please consider this post a unique opportunity to begin discussion of this important issue and facilitate further debate in the near future.
LessWrong gaming community
by TwistingFingers — 2011-09-26T02:15:09.675Z

Many of us express ourselves predominately through electronic games. As such, I feel that this aspect of our lives should be shared among our fellow gamers in the LessWrong community.

Video games are a great way to reduce compartmentalization and learn real-world rationality skills. Indeed, what brings us together at LessWrong can often be our love of games; someone without this advantage will probably find rationality difficult. In this light, outreach into the transhumanist/rationalist community to promote gaming is low-hanging fruit for serving the future of humanity.

In particular MMORPGs such as World of Warcraft, Rift, Runescape, and Maplestory are excellent beginnings on one's journey to a rationalist enlightenment.

Please consider this post a unique opportunity to begin discussion of this important issue and facilitate further debate in the near future.
How Likely Is Cryonics To Work?
by jefftk — 2011-09-25T23:38:04.599Z

If an American signs up for cryonics and pays their ~$300/year, what are their odds of being revived? Talking to people at LessWrong meetups I've heard estimates of 1 in 2. My friend George Dahl, whose opinion I respect a lot, guesses "less than 1 in 10^6". Niether has given me reasons, those numbers are opaque. My estimate of these odds pretty much determines whether I should sign up. I could afford $300/year, and I would if I thought the odds were 1:2, but not if they were 1:10^6. [1]

In order to see how likely this is to work, we should look at the process. I would sign up with a cryonics company and for life insurance. I'd go on living, enjoying my life and the people around me, paying my annual fees, until some point when I died. After death they would drain my blood, replace it with something that doesn't rupture cell walls when it freezes, freeze me in liquid nitrogen, and leave me there for a long time. At some point, probably after the development of nanotechnology, people would revive me, probably as a computer program.

There's a lot of steps there, and it's easy to see ways they could go wrong. [3] Let's consider some cases and try to get probabilities [4]:

Update: the probabilities below are out of date, and only useful for understanding the comments. I've made a spreadsheet listing both my updated probabilities and those for as many other people as I can find: https://docs.google.com/spreadsheet/...

0.03
You mess up the paperwork, either for cryonics or life insurance

0.10
Something happens to you financially where you can no longer afford this

0.06
You die suddenly or in a circumstance where you would not be able to be frozen in time (see leading causes of death)

0.04
You die of something like Altzheimers where the brain is degraded at death (Altzheimers is much more common than brain cancer)

0.01
The cryonics company is temporarily out of capacity and cannot actually take you, perhaps because lots of people died at once

0.02
The life insurance company does not pay out, perhaps it's insolvent, perhaps it argues you're not dead yet

0.02
You die in a hospital that refuses access to you by the cryonics people

0.02
After death your relatives reject your wishes and don't let the cryonics people freeze you

0.10
Some law is passed that prohibits cryonics (before you're even dead)

0.20
T

...
Guardian Angels: Discrete Extrapolated Volitions
by lessdazed — 2011-09-25T22:51:44.719Z

Questions for discussion, with my tentative answers. Assuming I am wrong about some things, there is something interesting to consider. This is inspired by the recent SL4-type and CEV-centric topics in the discussion section.

Questions:

I

Is it easier to calculate the extrapolated volition of an individual or a group?

If it is easier to do for an individual, is it because it is strictly simpler to do it, in that calculating humanity's CEV involves making at least every calculation that would be made for calculating the extrapolated volition of one individual?

How definitively can these questions be answered without knowing exactly how to calculate CEV?

II

Is it possible to create multiple AIs such that one AI does not prevent others from being created, such as by releasing equally powerful AIs simultaneously?

Is it possible to box AIs such that they reliably never escape before a certain, if short, period of time, such as by giving them a low-cost way out with a calculable minimum and maximum time to exploit that route?

Is it likely there would be a cooperative equilibrium among unmerged AIs?

III

Assuming the possibility of all of the following: what would happen if every person had a superintelligent AI with a utility function of that person's idealized extrapolated utility function?

How would that compare to a scenario with a single AI embodying a successful calculation of CEV?

What would be different if a person or some few people did not have a superintelligence valuing what they would value, and only many people had their own AI?

My Answers:

I

It depends on the error level tolerated. If only very low error is tolerated, it is easier to do it for a group.

N/A

Not sure.

II

Probably not.

Maybe, probably not, but impossible to know with high confidence.

Probably not. Throughout history, offense has often been a step ahead of defense, which often catches up to it. I think this is not particular to evolutionary biology or the technologies that happen to have been developed. It seems easier to break complicated things with many moving parts than to build and defend them. Also, specific technologies people plausibly speculate may exist are more powerful offensively than defensively. I would expect them to merge, probably peacefully.

III

Hard to say, as that would be trying to predict the actions of more intelligent be

...
A boltzmann brain question.
by DuncanS — 2011-09-25T22:35:55.354Z

I have a question on Boltzmann brains which I'd like to hear your opinions on - mostly because I don't know the answer to this....

First of all - a Boltzmann brain is the idea that - in a sufficiently large and long-lived universe, a brain just like mine or yours will occasionally pop into existence by sheer fluke. It doesn't happen very often - in fact it happens very, very infrequently. The brain will go on to have an experience or two before ceasing to exist on the grounds that it's surrounded by vacuum and cold - which is not a very good place for a disembodied brain to be.

Such a brain would have a very short life. Well, by a greater fluke, some of them might last for a longer time, but the balance of probabilities is that most boltzmann brains that think they had a long life merely had a lot of false memories of this life, planted by the fluke of their sudden pop into existence. And in their few seconds of life, they never got to realise that they didn't actually live that life, and their memories make no sense.

Well, boltzmann brains don't pop into existence by fluke very often - in the whole life of the observable universe it's overwhelmingly likely that it's never happened.

What might be more likely to happen? Well, you could have half a Boltzmann brain instead, and by sheer fluke, have the nerves leading from that half-brain stimulated as if the other half was there during the few seconds of the half-brain's life. This is still extremely unlikely, but tremendously more likely than having a whole Boltzmann brain appear. And the half-brain still thinks it has the same experience as before.

There is of course nothing to stop us from continuing this. Suppose we have a one-quarter brain? Much more probable. One millionth? Even more probable. Maybe even single elements of a nerve cell? More probable still. The smaller the piece is, the less of a fluke you need for it to come into existence, and the less of a fluke you need to continue to supply all the same inputs that it would have had in the event that the whole brain appeared.

So we keep dividing down until we hit the opposite problem. As we divide and divide the Boltzmann brain, all the time ensuring that the part we had left had the same experience as before, we eventually get down to really simple processes that we can easily simulate, and which happen in the real universe all the time. And inputs which ...
Convincing my dad to sign up for Alcor: Advice?
by EphemeralNight — 2011-09-25T22:01:54.668Z

Since it came to my attention that signing up for cryonics is not as pointless as I'd once thought, I've been pondering how to sell my dad on the idea.

This is somewhat urgent for a couple of reasons. First, he's already pushing sixty and would meet increased resistance in acquiring another life insurance policy at a much steeper rate than myself. Second, even given his age, he could afford to sign us both up easily, and after some consideration, convincing him seems like the more efficient path to being signed up myself than trying to arrange it for myself alone. And, well, he's my dad, and while he has his flaws, he's kind of awesome.

The sticking point, I can easily predict, will be getting passed his Cached Skepticism towards the concept of Cryonics. He is proud of being Skeptical; it's important to his self-concept, so I need to hit him with an opening that he can't easily dismiss. I would predict that if I simply linked him to Alcor's webpage, absolutely nothing would happen. I need something that will motivate him to investigate.

The frustrating part, is that I'm quite sure that if he'd never heard of cryonics before, he'd have no resistance to the idea beyond the basic pain of dispelling the emotional numbness towards a Certain Doom by suggesting it might not be certain after all. Unless my model of him is very much inaccurate, his Cached Skepticism of Cryonics is the only notable obstacle. I would appreciate any recommendations on which if any articles in particular would be best for initially getting through that.
Ideas for heuristics and biases research topic?
by Tesseract — 2011-09-25T18:20:53.170Z

Hey Less Wrong,

I'm currently taking a cognitive psychology class, and will be designing and conducting a research project in the field — and I'd like to do it on human judgment, specifically heuristics and biases. I'm currently doing preliminary research to come up with a more specific topic to base my project on, and I figured Less Wrong would be the place to come to find questions about flawed human judgment. So: any ideas?

(I'll probably be using these ideas mostly as guidelines for forming my research question, since I doubt it would be academically honest to take them outright. The study will probably take the form of a questionnaire or online survey, but experimental manipulation is certainly possible and it might be possible to make use of other psych department resources.)
What if we make better decisions when we trust our gut instincts? [Link]
by XiXiDu — 2011-09-25T12:22:45.033Z

[...]

How does one navigate a world of seemingly infinite alternatives? For thousands of years, the answer has seemed obvious: when faced with a difficult dilemma, we should carefully assess our options and spend a few moments consciously deliberating the information. Then, we should choose the toothpaste that best fits our preferences. This is how we maximize utility and get the most bang for the buck. We are rational agents – we should make decisions in a rational manner.

But what if rationality backfires? What if we make better decisions when we trust our gut instincts? While there is an extensive literature on the potential wisdom of human emotion, it’s only in the last few years that researchers have demonstrated that the emotional system (aka Type 1 thinking) might excel at complex decisions, or those involving lots of variables. If true, this would suggest that the unconscious is better suited for difficult cognitive tasks than the conscious brain, that the very thought process we’ve long disregarded as irrational and impulsive might actually be “smarter” than reasoned deliberation. This is largely because the unconscious is able to handle a surfeit of information, digesting the facts without getting overwhelmed. (Human reason, in contrast, has a very strict bottleneck and can only process about four bits of data at any given moment.)

[...]

The most widely cited demonstration of this theory is a 2006 Science paper led by Ap Dijksterhuis. (I wrote about the research in How We Decide.)

[...]

Dijksterhuis found that people given time to think in a rational manner – they could carefully contemplate each alternative – now chose the ideal car less than 25 percent of the time. In other words, they performed worse than random chance. However, subjects who were distracted for a few minutes found the best car nearly 60 percent of the time.

[...]

While the Dijksterhuis work generated plenty of buzz, it failed several tests of replication. This led many researchers to suggest that the supposed benefit of unconscious thought were an experimental accident, or perhaps a side-effect of incubation. [...] However, a new paper, published this month in Emotion by scientists then at Cornell University, provided the best test yet of the possible advantages of using our emotions to make complex decisions.

[...]

Once again, the “detail-

...
Meetup : Fort Collins, Colorado Meetup Wednesday Sept 28 7pm
by EvelynM — 2011-09-25T11:48:23.899Z

Discussion article for the meetup : Fort Collins, Colorado Meetup Wednesday Sept 28 7pm

WHEN:
28 September 2011 07:00:00PM (-0600)

WHERE:
Bean Cycle, 144 North College Avenue, Fort Collins, CO 80524

Coffee and chat with smart, interesting people.

Discussion article for the meetup : Fort Collins, Colorado Meetup Wednesday Sept 28 7pm
SIAI vs. FHI achievements, 2008-2010
by Kaj_Sotala — 2011-09-25T11:42:55.336Z

After reading the FHI achievement report for 2008-2010, I thought it might be useful to compare their achievements to those of SIAI during the same time period. Since SIAI does not have an equivalent report, I've mostly pulled the data of their achievements from the SIAI blog.

My intention here is to help figure out which organization makes better use of my donations. For that purpose, I'm only looking at actual concrete outputs, and ignoring achievements such as successful fundraising drives or the hiring of extra staff.

For citation counts, I'm using Google Scholar data as-is. Note that this will include both self-cites and some cites from pages that really shouldn't be counted, since Google Scholar seems to be a bit liberal about what it includes in its database. I'm unsure as to whether or not the citation counts are very meaningful, since there hasn't been much time for anyone to cite papers published in 2010, say. But I'm including them anyway.

Future of Humanity Institute

Publications. The Achievement Report highlights three books and 22 journal articles. In addition, FHI staff has written 34 book chapters for academic volumes, including Companion to Philosophy of Technology; New Waves in Philosophy of Technology; Philosophy: Theoretical and Empirical Explorations; and Oxford Handbook of Neuroethics.

The three are the hardcover and paperback editions of Human Enhancement, as well as a paperback edition of Anthropic Bias: Observation Selection Effects in Science and Philosophy. Human Enhancement has been cited 22 times. Anthropic Bias was originally published in 2002, so I'm not including its citation count.

The highlighted 22 journal articles had been cited 59 times in total. The overwhelmingly most cited article was Cognitive Enhancement: Methods, Ethics, Regulatory Challenges in Science and Engineering Ethics, with 39 cites. The runner-up was Probing the Improbable: Methodological. Challenges for Risks with Low Probabilities and High Stake, with 5 cites. The remaining articles had 0-3 cites. But while Cognitive Enhancement is listed as a 2009 paper, it's worth noting that the first draft version of it was posted on Nick Bostrom's website back in 2006, and it has had time to accumulate cites since then. If we exclude it, FHI's 2008-2010 papers have been cited 20 times.

It's not listed in the Achievement Report, but I also want to include the 2008 Whole Brai...
Cognitive Neuroscience, Arrow's Impossibility Theorem, and Coherent Extrapolated Volition
by lukeprog — 2011-09-25T11:15:25.095Z

Suppose we want to use the convergence of humanity's preferences as the utility function of a seed AI that is about to determine the future of its light cone.

We figured out how to get an AI to extract preferences from human behavior and brain activity. The AI figured out how to extrapolate those values. But my values and your values and Sarah Palin's values aren't fully converging in the simulation running the extrapolation algorithm. Our simulated beliefs are converging because on the path to reflective equilibrium our partially simulated selves have become true Bayesians and Aumann's Agreement Theorem holds. But our preferences aren't converging quite so well.

What to do? We'd like the final utility function in the FOOMed AI to adhere to some common-sense criteria:

Non-dictatorship: No single person's preferences should dictate what the AI does. Its utility function must take multiple people's (extrapolated) preferences into account.

Determinism: Given the same choices, and the same utility function, the AI should always make the same decisions.

Pareto efficiency: If every (extrapolated) person prefers action A to action B, the AI should prefer A to B.

Independence of irrelevant alternatives: If we — a group of extrapolated preference-sets — prefer A to B, and a new option C is introduced, then we should still prefer A to B regardless of what we think about C.

Now, Arrow's impossibility theorem says that we can only get the FOOMed AI's utility function to adhere to these criteria if the extrapolated preferences of each partially simulated agent are related to each other cardinally ("A is 2.3x better than B!") instead of ordinally ("A is better than B, and that's all I can say").

Now, if you're an old-school ordinalist about preferences, you might be worried. Ever since Vilfredo Pareto pointed out that cardinal models of a person's preferences go far beyond our behavioral data and that as far as we can tell utility has "no natural units," some economists have tended to assume that, in our models of human preferences, preference must be represented ordinally and not cardinally.

But if you're keeping up with the latest cognitive neuroscience, you might not be quite so worried. It turns out that preferences are encoded cardinally after all, and they do have a natural unit: action potentials per second. With cardinally encode...