trentmkelly/LessWrong-43k huggingface dataset:
How does real world expected utility maximization work?
by XiXiDu — 2012-03-09T11:20:07.453Z

I would like to ask for help on how to use expected utility maximization, in practice, to maximally achieve my goals.

As a real world example I would like to use the post 'Epistle to the New York Less Wrongians' by Eliezer Yudkowsky and his visit to New York.

How did Eliezer Yudkowsky compute that it would maximize his expected utility to visit New York?

It seems that the first thing he would have to do is to figure out what he really wants, his preferences1, right? The next step would be to formalize his preferences by describing it as a utility function and assign a certain number of utils2 to each member of the set, e.g. his own survival. This description would have to be precise enough to figure out what it would mean to maximize his utility function.

Now before he can continue he will first have to compute the expected utility of computing the expected utility of computing the expected utility of computing the expected utility3 ... and also compare it with alternative heuristics4.

He then has to figure out each and every possible action he might take, and study all of their logical implications, to learn about all possible world states he might achieve by those decisions, calculate the utility of each world state and the average utility of each action leading up to those various possible world states5.

To do so he has to figure out the probability of each world state. This further requires him to come up with a prior probability for each case and study all available data. For example, how likely it is to die in a plane crash, how long it would take to be cryonically suspended from where he is in case of a fatality, the crime rate and if aliens might abduct him (he might discount the last example, but then he would first have to figure out the right level of small probabilities that are considered too unlikely to be relevant for judgment and decision making).

I probably miss some technical details and got others wrong. But this shouldn't detract too much from my general request. Could you please explain how Less Wrong style rationality is to be applied practically? I would also be happy if you could point out some worked examples or suggest relevant literature. Thank you.

I also want to note that I am not the only one who doesn't know how to actually apply what is being discussed on Less Wrong in practice. From the comments:

You can’t believe in the impl

...
[SEQ RERUN] Amazing Breakthrough Day: April 1st
by MinibearRex — 2012-03-09T04:58:01.180Z

Today's post, Amazing Breakthrough Day: April 1st was originally published on 25 March 2008. A summary (taken from the LW wiki):

A proposal for a new holiday, in which journalists report on great scientific discoveries of the past as if they had just happened, and were still shocking.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was The Beauty of Settled Science, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Meetup : Fort Collins Meetup Saturday 17th
by EvelynM — 2012-03-09T03:46:20.911Z

Discussion article for the meetup : Fort Collins Meetup Saturday 17th

WHEN:
17 March 2012 05:00:00PM (-0700)

WHERE:
144 North College Avenue, Fort Collins, CO 80524

As requested, we're trying a weekend night to shake things up.

Meet down town for coffee, then back to my house for dinner, games and extended chatting.

Discussion article for the meetup : Fort Collins Meetup Saturday 17th
A taxonomy of Oracle AIs
by lukeprog — 2012-03-08T23:14:16.293Z

Sources: An old draft on Oracle AI from Daniel Dewey, conversation with Dewey and Nick Beckstead. See also Thinking Inside the Box and Leakproofing the Singularity.

Can we just create an Oracle AI that informs us but doesn't do anything?

"Oracle AI" has been proposed in many forms, but most proposals share a common thread: a powerful AI is not dangerous if it doesn't "want to do anything", the argument goes, and therefore, it should be possible to create a safe "Oracle AI" that just gives us information. Here, we discuss the difficulties of a few common types of proposed Oracle AI.

Two broad categories can be treated separately: True Oracle AIs, which are true goal-seeking AIs with oracular goals, and Oracular non-AIs, which are designed to be "very smart calculators" instead of goal-oriented agents.

True Oracle AIs

A True Oracle AI is an AI with some kind of oracular goal. Informally proposed oracular goals often include ideas such as "answer all questions", "only act to provide answers to questions", "have no other effect on the outside world", and "interpret questions as we would wish them to be interpreted.” Oracular goals are meant to "motivate" the AI to provide us with the information we want or need, and to keep the AI from doing anything else.

First, we point out that True Oracle AI is not causally isolated from the rest of the world. Like any AI, it has at least its observations (questions and data) and its actions (answers and other information) with which to affect the world. A True Oracle AI interacts through a somewhat low-bandwidth channel, but it is not qualitatively different from any other AI. It still acts autonomously in service of its goal as it answers questions, and it is realistic to assume that a superintelligent True Oracle AI will still be able to have large effects on the world.

Given that a True Oracle AI acts, by answering questions, to achieve its goal, it follows that True Oracle AI is only safe if its goal is fully compatible with human values. A limited interaction channel is not a good defense against a superintelligence.

There are many ways that omission of detail about human value could cause a "question-answering" goal to assign utility to a very undesirable state of the world, resulting in a undesirable future. A designer of an oracular goal must be certain to include a virtually endless list of qualifiers a...
Meetup : Atlanta
by hankx7787 — 2012-03-08T17:35:17.659Z

Discussion article for the meetup : Atlanta

WHEN:
17 March 2012 06:30:00PM (-0400)

WHERE:
2094 North Decatur Road, Decatur, GA 30033-5367

WHEN: 17 March 2012 06:30:00PM (-0500)

WHERE: 2094 North Decatur Road, Decatur, GA 30033-5367

The next meetup will be Saturday, March 17th at 6:30pm at Chocolate Coffee in Decatur:

http://www.mychocolatecoffee.com/

2094 North Decatur Road, Decatur, GA 30033-5367

(404) 982-0790

We will be finishing up the "Mysterious Answers to Mysterious Questions" sequence at the next meeting. As always, any other topics you want to bring up are fair game!

Here is the official agenda of our next meeting:

http://wiki.lesswrong.com/wiki/Mysterious_Answers_to_Mysterious_Questions

1.26 "Science" as Curiosity-Stopper

1.27 Applause Lights

1.28 Truly Part of You

1.29 Chaotic Inversion

Please let me know if you have any questions or comments! I look forward to seeing everyone there!

ps. join the mailing list! http://groups.google.com/group/atlanta-less-wrong-meetup-group

Discussion article for the meetup : Atlanta
Conjunction fallacy and probabilistic risk assessment.
by Dmytry — 2012-03-08T15:07:13.934Z

Summary:

There is a very dangerous way in which conjunction fallacy can be exploited. One can present you with 2..5 detailed, very plausible failure scenarios whose probabilities are shown to be very low, using solid mathematics; then if you suffer from conjunction fallacy, it will look like this implies high safety of a design - while in fact it's the detailedness of the scenario that makes probability so low.

Even if you realize that there may be many other scenarios that were not presented to you, you still have an incredibly low probability number on a highly plausible ("most likely") failure scenario, which you, being unaware of the powers of conjunction, attribute to safety of the design.

The conjunction fallacy can be viewed as poor understanding of relation between plausibility and probability. Addition of extra details doesn't make scenario seem less plausible (it can even increase plausibility), but does mathematically make it less probable.

Details:

What happens if a risk assessment is being prepared for (and possibly by) sufferers of conjunction fallacy?

Detailed example scenarios will be chosen, such as:

A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.

Then as a risk estimate, you multiply probability of Russian invasion of Poland, by probability of it resulting in suspension of diplomatic relations between US and SU, and multiply by probability of it happening specifically in 1983 . The resulting probability could be extremely small for sufficiently detailed scenario (you can add the polish prime minister being assassinated if your probability is still too high for comfort).

To a sufferer of conjunction fallacy it looks like a very plausible, 'most likely' scenario has been shown highly improbable, and thus the risks are low. The sufferer of conjunction fallacy does not expect that this probability could be very low in unsafe design.

It seems to me that the risk assessment is routinely done in such a fashion. Consider Space Shuttle's reliability, or the NRC cost-benefit analyses for the spent fuel pools , which goes as low as one in 45 millions years for the most severe scenario. (Same seem to happen in all of the NRC resolutions, to varying extent; feel free to dig through)

Those reports looked outright insane to me - a very small number of highly detailed scena...
James Martin Postdoctoral Research Fellowship: Socio-economic Impacts of Technological Change
by Stuart_Armstrong — 2012-03-08T14:54:27.122Z

We are pleased to announce a new vacancy at the Programme on the Impacts of Future Technology. Please forward to any who would be interested.

James Martin Postdoctoral Research Fellowship:

Socio-economic Impacts of Technological Change
with the Programme on the Impacts of Future Technology

University of Oxford

Faculty of Philosophy

The Future of Humanity Institute, Oxford Martin School

Grade 7: £29,249–£39,257 per annum
Protocol reference number: HUM/11043F/E

Applications are invited for a Research Fellowship within the Oxford Martin Programme on the Impacts of Future Technology, an interdisciplinary programme within the Oxford Martin School at Oxford University. This Fellowship is available on a one year full-time or two years part-time fixed term basis.

The Programme, directed by Professor Nick Bostrom, analyzes possibilities related to long-range technological change and potential social impacts of future transformative technologies. Research foci include the future of computing and machine intelligence, existential risks, predictive and evaluative uncertainty, and related philosophical issues.

The postholder will conduct research on socio-economic and strategic impacts of potentially transformative or disruptive future technological innovations, including (but not limited to) advances in computing and machine intelligence, biosecurity and surveillance technology. Academic background is open. Potential areas include economics, political science, legal theory, and sociology; other relevant areas include environmental economics, game theory, and risk management. A multidisciplinary background would be favourable.

For further particulars and application details, please see:

http://www.futuretech.ox.ac.uk/vacancies

or contact:futuretech@philosophy.ox.ac.uk.

The deadline for applications is Monday 9th April.
Sudden Future Singularity (SFS) as soon as 8.7 million years in the future?
by MichaelAnissimov — 2012-03-08T10:41:23.236Z

No, not the kind of Singularity usually discussed here... I'm referring to the possibility of phantom energy-driven rips in the cosmos caused by accelerating expansion, or "sudden future singularities of pressure". (Technically: "a momentary infinite peak in the tidal forces of the universe.") A recent paper by Ghodsi & Hendri shows that cosmic microwave background, baryon acoustic oscillations (BAO), and type 1a supernovae data is consistent with the possibility of a sudden future singularity as soon as 8.7 million years from now.

"Cosmological tests of sudden future singularities"
http://arxiv.org/pdf/1201.6661v1.pdf

As I understand it, the authors are not saying that a SFS is likely 8.7 million years from now, just possible. This puts a dampener on the notion that the only plausible scenario of cosmological breakdown is Heat Death.

Here's another paper that outlines other exotic cosmological singularities which have been under discussion in the cosmology community for the past decade, and the behavior of pointlike particles and strings as they approach such singularities.
I Was Not Almost Wrong But I Was Almost Right: Close-Call Counterfactuals and Bias
by Kaj_Sotala — 2012-03-08T05:39:04.336Z

Abstract: "Close-call counterfactuals", claims of what could have almost happened but didn't, can be used to either defend a belief or to attack it. People have a tendency to reject counterfactuals as improbable when those counterfactuals threaten a belief (the "I was not almost wrong" defense), but to embrace counterfactuals that support a belief (the "I was almost right" defense). This behavior is the strongest in people who score high on a test for need for closure and simplicity. Exploring counterfactual worlds can be used to reduce overconfidence, but it can also lead to logically incoherent answers, especially in people who score low on a test for need for closure and simplicity.

”I was not almost wrong”

Dr. Zany, the Nefarious Scientist, has a theory which he intends to use to achieve his goal of world domination. ”As you know, I have long been a student of human nature”, he tells his assistant, AS-01. (Dr. Zany has always wanted to have an intelligent robot as his assistant. Unfortunately, for some reason all the robots he has built have only been interested in eradicating the color blue from the universe. And blue is his favorite color. So for now, he has resorted to just hiring a human assistant and referring to her with a robot-like name.)

”During my studies, I have discovered the following. Whenever my archnemesis, Captain Anvil, shows up at a scene, the media will very quickly show up to make a report about it, and they prefer to send the report live. While this is going on, the whole city – including the police forces! - will be captivated by the report about Captain Anvil, and neglect to pay attention to anything else. This happened once, and a bank was robbed on the other side of the city while nobody was paying any attention. Thus, I know how to commit the perfect crime – I simply need to create a diversion that attracts Captain Anvil, and then nobody will notice me. History tells us that this is the inevitable outcome of Captain Anvil showing up!”

But to Dr. Zany's annoyance, AS-01 is always doubting him. Dr. Zany has often considered turning her into a brain-in-a-vat as punishment, but she makes the best tuna sandwiches Dr. Zany has ever tasted. He's forced to tolerate her impundence, or he'll lose that culinary pleasure.

”But Dr. Zany”, AS-01 says. ”Suppose that some TV reporter...
[SEQ RERUN] The Beauty of Settled Science
by MinibearRex — 2012-03-08T04:24:18.223Z

Today's post, The Beauty of Settled Science was originally published on 24 March 2008. A summary (taken from the LW wiki):

Most of the stuff reported in Science News is false, or at the very least, misleading. Scientific controversies are topics of such incredible difficulty that even people in the field aren't sure what's true. Read elementary textbooks. Study the settled science before you try to understand the outer fringes.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was If You Demand Magic, Magic Won't Help, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Meetup : Pittsburgh Meetup: Big Gaming Fun 4!
by Kenoubi — 2012-03-08T02:29:51.999Z

Discussion article for the meetup : Pittsburgh Meetup: Big Gaming Fun 4!

WHEN:
11 March 2012 12:00:00PM (-0500)

WHERE:
1324 Wightman St., Pittsburgh, PA 15217

You can see my game collection here; please bring anything else you'd like to play. We can order food and go as late as 19:00. If I get paged I may have to deal with an emergency (from home, using my laptop), but if that doesn't bother you, it doesn't bother me. I have a cat. Please let me know if you're allergic and need me to put her upstairs. RSVP here or by sending me a private message (but don't not show up because you didn't RSVP, I just want a rough idea of the number of attendees). Ring the bell, knock, or call or text (412) 657-1395 to get in when you get there. I intend to hold these every 2-3 weeks, so watch this space!

Discussion article for the meetup : Pittsburgh Meetup: Big Gaming Fun 4!
Making computer systems with extended Identity 
by  — 2012-03-08T01:29:04.957Z

We often assume that an AI will have an identity and goals of its own. That it will be some separate entity from a human being or group of humans.

In physics there are no separate entities, merely a function evolving through time. So any identity needs to be constructed by systems within physics, and the boundaries are arbitrary. We have been built by evolution and all the cells in our body have the same programming so we have a handy rule of thumb that our body is "us" as it is created by a single replicating complex. So we assume that a computational entity, if it develops a theory of self, will only include its processing elements or code and nothing else in its notion of identity. But what an system identifies with can be controlled and specifed.

If a system identifies a human as an important part of itself it will strive to protect it and its normal functioning, as we instinctively protect important parts of ourselves such as the head and genitals.

So what possible objections to this are there?

1) Humans are spatially separate from the machine so they won't consider it part of themselves

We have a habit of identifying with groups larger than our own, such as countries and integrating our goals with theirs to different extents. Spatial co-location is not required.

2) Humans are very different from computers they will see them as "other"

Different parts of the human body are very diverse, but all of it is seen as a singular entity. Spleen and all.

3) A human will do things the computer doesn't know why, so It will not see it as part of itself.

Self-knowledge is not required for self-identification. Different parts of the brain are black boxes to others, we make up explanations for why we do things, in cases like blind sight, so there is no need for all the parts of the system to be self-reflective.

So can we make advanced computational systems that consider humanity as part of them? One possible problem with this approach is that if it doesn't get information from a part of humanity, it may well ignore its needs even if it still considers it part of itself. So perhaps it could bond with a single human being and have a myriad of the systems and rely on negotiations between them to [form a balance](http://lesswrong.com/r/discussion/lw/a7y/friendly_ai_society/).

One thing to consider is that some people only consider the program encoded by their neural patterns to be t...
Meetup : Queueing and More
by SarahNibs — 2012-03-07T20:44:36.693Z

Discussion article for the meetup : Queueing and More

WHEN:
11 March 2012 03:00:00PM (-0800)

WHERE:
15207 NE 72nd St, Redmond, WA 98052

Discussion about reducing queue size as a heuristic, arbitrary discussion, dinner, boxing probably not, board games maybe, alcohol likely.

See http://groups.google.com/group/lw-seattle/browse_thread/thread/f3bac3bc1b505f7b for details and ride planning.

Discussion article for the meetup : Queueing and More
Friendly AI Society
by Douglas_Reay — 2012-03-07T19:31:23.052Z

Summary: AIs might have cognitive biases too but, if that leads to it being in their self-interest to cooperate and take things slow, that might be no bad thing.

The value of imperfection

When you use a traditional FTP client to download a new version of an application on your computer, it downloads the entire file, which may be several gig, even if the new version is only slightly different from the old version, and this can take hours.

Smarter software splits the old file and the new file into chunks, then compares a hash of each chunk, and only downloads those chunks that actually need updating. This 'diff' process can result in a much faster download speed.

Another way of increasing speed is to compress the file. Most files can be compressed a certain amount, without losing any information, and can be exactly reassembled at the far end. However, if you don't need a perfect copy, such as with photographs, using lossy compression can result in very much more compact files and thus faster download speeds.

Cognitive misers

The human brain likes smart solutions. In terms of energy consumed, thinking is expensive, so the brain takes shortcuts when it can, if the resulting decision making is likely to be 'good enough' in practice. We don't store in our memories everything our eyes see. We store a compressed version of it. And, more than that, we run a model of what we expect to see, and flick our eyes about to pick up just the differences between what our model tells us to expect to see, and what is actually there to be seen. We are cognitive misers

When it comes to decision making, our species generally doesn't even try to achieve pure rationality. It uses bounded rationality, not just because that's what we evolved, but because heuristics, probabilistic logic and rational ignorance have a higher marginal cost efficiency (the improvements in decision making don't produce a sufficient gain to outweigh the cost of the extra thinking).

This is why, when pattern matching (coming up with causal hypotheses to explain observed correlations), are our brains designed to be optimistic (more false positives than false negatives). It isn't just that being eaten by a tiger is more costly than starting at shadows. It is that we can't afford to keep all the base data. If we ...