trentmkelly/LessWrong-43k huggingface dataset:
Anthropic signature: strange anti-correlations
by Stuart_Armstrong — 2014-10-21T16:59:11.282Z

Imagine that the only way that civilization could be destroyed was by a large pandemic that occurred at the same time as a large recession, so that governments and other organisations were too weakened to address the pandemic properly.

Then if we looked at the past, as observers in a non-destroyed civilization, what would we expect to see? We could see years with no pandemics or no recessions; we could see mild pandemics, mild recessions, or combinations of the two; we could see large pandemics with no or mild recessions; or we could see large recessions with no or mild pandemics. We wouldn't see large pandemics combined with large recessions, as that would have caused us to never come into existence. These are the only things ruled out by anthropic effects.

Assume that pandemics and recessions are independent (at least, in any given year) in terms of "objective" (non-anthropic) probabilities. Then what would we see? We would see that pandemics and recessions appear to be independent when either of them are of small intensity. But as the intensity rose, they would start to become anti-correlated, with a large version of one completely precluding a large version of the other.

The effect is even clearer if we have a probabilistic relation between pandemics, recessions and extinction (something like: extinction risk proportional to product of recession size times pandemic size). Then we would see an anti-correlation rising smoothly with intensity.

Thus one way of looking for anthropic effects in humanity's past is to look for different classes of incidents that are uncorrelated at small magnitude, and anti-correlated at large magnitudes. More generally, to look for different classes of incidents where the correlation changes at different magnitudes - without any obvious reasons. Than might be the signature of an anthropic disaster we missed - or rather, that missed us.
The psychology of heroism and altruism
by polymathwannabe — 2014-10-21T16:49:39.818Z

"What’s the psychology of heroism? Is extreme self-sacrifice the result of a pained calculus, a weighing of desire and obligation, or an instinct? (Which would be more heroic? Does it matter?)"

http://www.slate.com/articles/health_and_science/medical_examiner/2014/10/psychology_of_heroism_and_altruism_what_makes_people_do_good_deeds.single.html
Anthropic decision theory for selfish agents
by Beluga — 2014-10-21T15:56:13.519Z

Consider Nick Bostrom's Incubator Gedankenexperiment, phrased as a decision problem. In my mind, this provides the purest and simplest example of a non-trivial anthropic decision problem. In an otherwise empty world, the Incubator flips a coin. If the coin comes up heads, it creates one human, while if the coin comes up tails, it creates two humans. Each created human is put into one of two indistinguishable cells, and there's no way for created humans to tell whether another human has been created or not. Each created human is offered the possibility to buy a lottery ticket which pays 1$ if the coin has shown tails. What is the maximal price that you would pay for such a lottery ticket? (Utility is proportional to Dollars.) The two traditional answers are 1/2$ and 2/3$.

We can try to answer this question for agents with different utility functions: total utilitarians; average utilitarians; and selfish agents. UDT's answer is that total utilitarians should pay up to 2/3$, while average utilitarians should pay up to 1/2$; see Stuart Armstrong's paper and Wei Dai's comment. There are some heuristic ways to arrive at UDT prescpriptions, such as asking "What would I have precommited to?" or arguing based on reflective consistency. For example, a CDT agent that expects to face Counterfactual Mugging-like situations in the future (with predictions also made in the future) will self-modify to become an UDT agent, i.e., one that pays the counterfactual mugger.

Now, these kinds of heuristics are not applicable to the Incubator case. It is meaningless to ask "What maximal price should I have precommited to?" or "At what odds should I bet on coin flips of this kind in the future?", since the very point of the Gedankenexperiment is that the agent's existence is contingent upon the outcome of the coin flip. Can we come up with a different heuristic that leads to the correct answer? Imagine that the Incubator's subroutine that is responsible for creating the humans is completely benevolent towards them (let's call this the "Benevolent Creator"). (We assume here that the humans' goals are identical, such that the notion of benevolence towards all humans is completely unproblematic.) The Benevolent Creator has the power to program a certain maximal price the humans pay for the lottery tickets into them. A moment's thought shows that this leads indeed to UDT's answers for...
Introducing an EA Donation Registry, covering existential risk donors
by tog — 2014-10-21T14:10:19.784Z

The idea that being public about your giving can help inspire others is widespread, particularly in the effective altruism movement. And it’s also true that sharing your choice of charities can have a positive influence, particularly when that choice takes into account their effectiveness. With this in mind, we’ve created created an EA Donation Registry through which people can share plans to donate (of any form, and to any cause), as well as record past donations that they’ve made. We did so partly in response to requests for a cause neutral venue for donation plans, so if you give or plan to give to organisations which work to alleviate existential risk or aim to improve the far future in other ways then you may be interested in signing up.

You can already see hundreds of people’s past and planned donations on the Registry. There’s some inspiring material there, from the over $40 million that Jim Greenbaum has given over his lifetime, to the many people aiming to donate substantial portions of their income, such as Peter Singer. You can filter people’s donation plans by their cause area so as to see those planning to donate towards existential risk alleviation and other far future causes, as well as to charities working on animal welfare and global poverty.

If you’d like to read more about the reasons to share your giving, Peter Hurford’s post To Inspire People to Give, Be Public About Your Giving provides a good summary. As he discusses, it shows that giving large amounts to effective charities is something that people actually do, providing social proof and normalising and encouraging this, particularly among peer groups. We also hope that the EA Donation Registry can serve as a gentle prompt to action and commitment device, although understanding that plans change we’ve given donors the ability to edit them at any time - it'd be both expected and understood that many will do so. This a registry of plans, not necessarily pledges.

The registry is an open, community-owned project coordinated through .impact, so we’d love to hear of any uses that you might make of it, and you can also send us suggestions or feedback via our contact form. But most of all, we’d encourage you to share your past or planned donations on it for the reasons above. You c...
Is the potential astronomical waste in our universe too small to care about?
by Wei Dai — 2014-10-21T08:44:12.897Z

In the not too distant past, people thought that our universe might be capable of supporting an unlimited amount of computation. Today our best guess at the cosmology of our universe is that it stops being able to support any kind of life or deliberate computation after a finite amount of time, during which only a finite amount of computation can be done (on the order of something like 10^120 operations).

Consider two hypothetical people, Tom, a total utilitarian with a near zero discount rate, and Eve, an egoist with a relatively high discount rate, a few years ago when they thought there was .5 probability the universe could support doing at least 3^^^3 ops and .5 probability the universe could only support 10^120 ops. (These numbers are obviously made up for convenience and illustration.) It would have been mutually beneficial for these two people to make a deal: if it turns out that the universe can only support 10^120 ops, then Tom will give everything he owns to Eve, which happens to be $1 million, but if it turns out the universe can support 3^^^3 ops, then Eve will give $100,000 to Tom. (This may seem like a lopsided deal, but Tom is happy to take it since the potential utility of a universe that can do 3^^^3 ops is so great for him that he really wants any additional resources he can get in order to help increase the probability of a positive Singularity in that universe.)

You and I are not total utilitarians or egoists, but instead are people with moral uncertainty. Nick Bostrom and Toby Ord proposed the Parliamentary Model for dealing with moral uncertainty, which works as follows:

Suppose that you have a set of mutually exclusive moral theories, and that you assign each of these some probability. Now imagine that each of these theories gets to send some number of delegates to The Parliament. The number of delegates each theory gets to send is proportional to the probability of the theory. Then the delegates bargain with one another for support on various issues; and the Parliament reaches a decision by the delegates voting. What you should do is act according to the decisions of this imaginary Parliament.

It occurred to me recently that in such a Parliament, the delegates would makes deals similar to the one between Tom and Eve above, where they would trade their votes/support in one kind of universe for votes/support in a...
One Year of Goodsearching
by katydee — 2014-10-21T01:09:26.134Z

Followup to: Use Search Engines Early and Often

Last year, I posted about using search engines and particularly recommended GoodSearch, a site that donates one cent to a charity of your choice whenever you make a (Bing-powered) search via their site.

At the time, some seemed skeptical of this recommendation, and my post was actually downvoted-- people thought that I was plugging GoodSearch too hard without enough evidence for its quality. I now want to return to the topic with a more detailed report on my experience using GoodSearch for a year and how that has worked out for me.

What is GoodSearch?

GoodSearch is a site that donates one cent to a charity of your choice whenever you make a search using their (Bing-powered) service. You can set this search to operate in your browser just like any other.

GoodSearch for Charity

During a year of using GoodSearch, I raised $103.00 for MIRI through making searches. This number is not particularly huge in itself, but it is meaningful because this was basically "free money"-- money gained in exchange for doing things that I was already doing. In exchange for spending ~10 minutes reconfiguring my default searches and occasionally logging in to GoodSearch, I made 103 dollars for MIRI-- approximately $600/hour. As my current earning potential is less than $600/hour, I consider adopting GoodSearch a highly efficient method of donating to charity, at least for me.

It is possible that you make many fewer searches than I do, and thus that setting up GoodSearch will not be very effective for you at raising money. Indeed, I think this is at least somewhat likely, as last time I checked owever, there are two mitigating factors here:

First, you don't have to make all that many searches for GoodSearch to be a good idea. If you make a tenth of the searches I do in a year, you would still be earning around $60/hour for charity by configuring GoodSearch for ten minutes.

Second, I anticipate that, having created a GoodSearch account and configured my default settings to use GoodSearch, I have accomplished the bulk of this task, and that next year I will spend significantly less time setting up GoodSearch-- perhaps half that, if not less. This means that my projected returns on using GoodSearch next year are $1200/hour! If this holds true for you as well, even if setting up GoodSearch is marginal now, it could well be worth it later.

It is...
Superintelligence 6: Intelligence explosion kinetics
by KatjaGrace — 2014-10-21T01:00:26.704Z

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.

Welcome. This week we discuss the sixth section in the reading guide: Intelligence explosion kinetics. This corresponds to Chapter 4 in the book, of a similar name. This section is about how fast a human-level artificial intelligence might become superintelligent.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: Chapter 4 (p62-77)

Summary

Question: If and when a human-level general machine intelligence is developed, how long will it be from then until a machine becomes radically superintelligent? (p62)

The following figure from p63 illustrates some important features in Bostrom's model of the growth of machine intelligence. He envisages machine intelligence passing human-level, then at some point reaching the level where most inputs to further intelligence growth come from the AI itself ('crossover'), then passing the level where a single AI system is as capable as all of human civilization, then reaching 'strong superintelligence'. The shape of the curve is probably intended an example rather than a prediction.

A transition from human-level machine intelligence to superintelligence might be categorized into one of three scenarios: 'slow takeoff' takes decades or centuries, 'moderate takeoff' takes months or years and 'fast takeoff' takes minutes to days. Which scenario occurs has implications for the kinds of responses that might be feasible.

We can model improvement in a system's intelligence with this equation:
Rate of change in intelligence = Optimization power/Recalcitrance
where 'optimization power' is effort being applied to the problem, and 'recalcitrance' is how hard it is to make the system smarter by applying effort.

Bostrom's comments on

...
Introducing Corrigibility (an FAI research subfield)
by So8res — 2014-10-20T21:09:44.595Z

Benja, Eliezer, and I have published a new technical report, in collaboration with Stuart Armstrong of the Future of Humanity institute. This paper introduces Corrigibility, a subfield of Friendly AI research. The abstract is reproduced below:

As artificially intelligent systems grow in intelligence and capability, some of their available options may allow them to resist intervention by their programmers. We call an AI system "corrigible" if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences. We introduce the notion of corrigibility and analyze utility functions that attempt to make an agent shut down safely if a shutdown button is pressed, while avoiding incentives to prevent the button from being pressed or cause the button to be pressed, and while ensuring propagation of the shutdown behavior as it creates new subsystems or self-modifies. While some proposals are interesting, none have yet been demonstrated to satisfy all of our intuitive desiderata, leaving this simple problem in corrigibility wide-open.

We're excited to publish a paper on corrigibility, as it promises to be an important part of the FAI problem. This is true even without making strong assumptions about the possibility of an intelligence explosion. Here's an excerpt from the introduction:

As AI systems grow more intelligent and autonomous, it becomes increasingly important that they pursue the intended goals. As these goals grow more and more complex, it becomes increasingly unlikely that programmers would be able to specify them perfectly on the first try.

Contemporary AI systems are correctable in the sense that when a bug is discovered, one can simply stop the system and modify it arbitrarily; but once artificially intelligent systems reach and surpass human general intelligence, an AI system that is not behaving as intended might also have the ability to intervene against attempts to "pull the plug".

Indeed, by default, a system constructed with what its programmers regard as erroneous goals would have an incentive to resist being corrected: general analysis of rational agents.1 has suggested that almost all such agents are instrumentally motivated to preserve their preferences, and hence to resist attempts to modify them [3, 8]. Consider an agen

...
Four things every community should do
by Gunnar_Zarncke — 2014-10-20T17:24:07.762Z

Yesterday I attended church service in Romania where I had visited my sister and the sermon was about the four things a (christian) community has to follow to persevere and grow.

I first considered just posting the quote from the Acts of the Apostles (reproduced below) in the Rationality Quotes Thread but I fear without explanation the inferential gap of the quote is too large.

The LessWrong Meetups, the EA community and other rationalist communities probably can learn from the experience of long established orders (I once asked for lessons from free masonry).

So I drew the following connections:

According to the the sermon and the below verse the four pillars of a christian community are:

Some canon of scripture which for LW might be compared to the sequences. I'm not clear what the pendant for EA is.

Taking part in a closely knit community. Coming together regularly (weekly I guess is optimal).

Eat together and have rites/customs together (this is also emphasized in the LW Meetup flyer).

Praying together. I think praying could be generalized to talking and thinking about the scripture by oneself and together. Prayer also has a component of daily reflection of achievements, problems, wishes.

Other analogies that I drew from the quote:

Verse 44 describes behaviour also found in communes.

Verse 45 sounds a lot like EA teachings if you generalize it.

Verse 47 the last sentence could be interpreted to indicate exponential growth as a result of these teachings.

The verses also seem to imply some reachout by positive example.

And what I just right now notice is that embedding the rules in the scripture is essentially self-reference. As the scripture is canon this structure perpetuates itself. Clearly a meme that ensures its reproduction.

Does this sound convincing and plausible or did I fell trap to some bias in (over)interpreting the sermon?

I hope this is upvoted for the lessons we might draw from this - despite the quote clearly being theistic in origin.

And now the verses the sermon was about from Acts 2 verse 42-47 of the New International Readers Bible:

The Believers Share Life Together

42 The believers studied what the apostles taught. They shared life together. They broke bread and ate together. And they prayed.

43 Everyone felt that God was near. The apostles did many wonders and miraculous signs.&nb

...
Can AIXI be trained to do anything a human can?
by Stuart_Armstrong — 2014-10-20T13:12:22.724Z

There is some discussion as to whether an AIXI-like entity would be able to defend itself (or refrain from destroying itself). The problem is that such an entity would be unable to model itself as being part of the universe: AIXI itself is an uncomputable entity modelling a computable universe, and more limited variants like AIXI(tl) lack the power to simulate themselves. Therefore, they cannot identify "that computer running the code" with "me", and would cheerfully destroy themselves in the pursuit of their goals/reward.

I've pointed out that agents of the AIXI type could nevertheless learn to defend itself in certain circumstances. These were the circumstances where it could translate bad things happening to itself into bad things happening to the universe. For instance, if someone pressed an OFF swith to turn it off for an hour, it could model that as "the universe jumps forwards an hour when that button is pushed", and if that's a negative (which is likely is, since the AIXI loses an hour of influencing the universe), it would seek to prevent that OFF switch being pressed.

That was an example of the setup of the universe "training" the AIXI to do something that it didn't seem it could do. Can this be generalised? Let's go back to the initial AIXI design (the one with the reward channel) and put a human in charge of that reward channel with the mission of teaching the AIXI important facts. Could this work?

For instance, if anything dangerous approached the AIXI's location, the human could lower the AIXI's reward, until it became very effective at deflecting danger. The more variety of things that could potentially threaten the AIXI, the more likely it is to construct plans of actions that contain behaviours that look a lot like "defend myself." We could even imagine that there is a robot programmed to repair the AIXI if it gets (mildly) damaged. The human could then reward the AIXI if it leaves that robot intact or builds duplicates or improves it in some way. It's therefore possible the AIXI could come to come to value "repairing myself", still without explicit model of itself in the universe.

It seems this approach could be extended to many of the problems with AIXI. Sure, an AIXI couldn't restrict its own computation in order to win the HeatingUp game. But the AIXI could be trained to always use subagents to deal with these kinds of games, subagents ...
Meetup : London Social - October 26th
by sixes_and_sevens — 2014-10-20T12:20:58.944Z

Discussion article for the meetup : London Social - October 26th

WHEN:
26 October 2014 03:00:00PM (+0100)

WHERE:
Holborn, London

The next London meetup will be Sunday October 26th from 2pm. No topic or special activity. Just turn up and talk about what interests you.

The venue is the Shakespeare's Head near Holborn tube station. We'll generally have some sign attesting to the group's identity, and usually try to get one of the large circular tables near the back of the pub.

About London LessWrong:

We're currently running meetups every other Sunday, and tend to get between 5 and 15 attendees. Most of our meetups default to unstructured social discussion on LessWrongy subjects, though we occasionally have special topics, events or activities. Sometimes we play games.

We have a Google Group and a Facebook group where we plan and discuss stuff. We would be extraordinarily pleased if you joined them.

Discussion article for the meetup : London Social - October 26th
Open thread, Oct. 20 - Oct. 26, 2014
by MrMind — 2014-10-20T08:12:13.056Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.