trentmkelly/LessWrong-43k huggingface dataset:
Patternist friendly AI risk
by bokov — 2013-09-12T13:00:11.432Z

It seems to me that most AI researchers on this site are patternists in the sense of believing that the anti-zombie principle necessarily implies:

1. That it will ever become possible *in practice* to create uploads or sims that are close enough to our physical instantiations that their utility to us would be interchangeable with that of our physical instantiations.

2. That we know (or will know) enough about the brain to know when this threshold is reached.

But, like any rationalists extrapolating from unknown unknowns... or heck, extrapolating from anything... we must admit that one or both of the above statements could be wrong without also making friendly AI impossible. What would be the consequences of such error?

I submit that one such consequence could be an FAI that is also wrong on these issues but not only do we fail to check for such a failure mode, it actually looks to us like what we would expect the right answer to look because we are making the same error.

If simulation/uploading really does preserve what we value about our lives then the safest course of action is to encourage as many people to upload as possible. It would also imply that efforts to solve the problem of mortality by physical means will at best be given an even lower priority than they are now, or at worst cease altogether because they would seem to be a waste of resources.

Result: people continue to die and nobody including the AI notices, except now they have no hope of reprieve because they think the problem is already solved.

Pessimistic Result: uploads are so widespread that humanity quietly goes extinct, cheering themselves onward the whole time

Really Pessimistic Result: what replaces humanity are zombies, not in the qualia sense but in the real sense that there is some relevant chemical/physical process that is not being simulated because we didn't realize it was relevant or hadn't noticed it in the first place.

Possible Safeguards:

* Insist on quantum level accuracy (yeah right)

* Take seriously the general scenario of your FAI going wrong because you are wrong in the same way and fail to notice the problem.

* Be as cautious about destructive uploads as you would be about, say, molecular nanotech.

* Make sure you knowledge of neuroscience is at least as good as you knowledge of computer science and decision theory before you advocate digit...
Meetup : Brussels monthly meetup
by Roxolan — 2013-09-12T08:53:22.133Z

Discussion article for the meetup : Brussels monthly meetup

WHEN: 14 September 2013 01:00:00PM (+0200)

WHERE: Rue des Alexiens 55 1000 Bruxelles

Brussels meetups now happen every month on the second Saturday of the month (even when we forget to announce them in time...).

We are once again meeting at 'La fleur en papier doré' close to the Brussels Central station. If you feel like an intelligent discussion and are in the neighborhood, consider dropping by. The meeting will be in English to facilitate both French and Dutch speaking members.

If you are coming for the first time, please consider filling out this one minute form, to share your contact information.

Discussion article for the meetup : Brussels monthly meetup
Meetup : Berlin: Fermi paradox discussion
by blob — 2013-09-12T08:27:33.984Z

Discussion article for the meetup : Berlin: Fermi paradox discussion

WHEN:
18 October 2013 07:00:00PM (+0200)

WHERE:
S Wuhletal, 12621 Berlin

We're meeting to discuss the Fermi paradox and the great filter. Everyone is welcome! There'll probably be 6-12 people.

Please look at the mailing list for details.

Discussion article for the meetup : Berlin: Fermi paradox discussion
How well will policy-makers handle AGI? (initial findings)
by lukeprog — 2013-09-12T07:21:30.255Z

Cross-posted from MIRI's blog.

MIRI's mission is "to ensure that the creation of smarter-than-human intelligence has a positive impact." One policy-relevant question is: How well should we expect policy makers to handle the invention of AGI, and what does this imply about how much effort to put into AGI risk mitigation vs. other concerns?

To investigate these questions, we asked Jonah Sinick to examine how well policy-makers handled past events analogous in some ways to the future invention of AGI, and summarize his findings. We pre-committed to publishing our entire email exchange on the topic (with minor editing), just as with our project on how well we can plan for future decades. The post below is a summary of findings from our full email exchange (.docx) so far.

As with our investigation of how well we can plan for future decades, we decided to publish our initial findings after investigating only a few historical cases. This allows us to gain feedback on the value of the project, as well as suggestions for improvement, before continuing. It also means that we aren't yet able to draw any confident conclusions about our core questions.

The most significant results from this project so far are:

We came up with a preliminary list of 6 seemingly-important ways in which a historical case could be analogous to the future invention of AGI, and evaluated several historical cases on these criteria.

Climate change risk seems sufficiently disanalogous to AI risk that studying climate change mitigation efforts probably gives limited insight into how well policy-makers will deal with AGI risk: the expected damage of climate change appears to be very small relative to the the expected damage due to AI risk, especially when one looks at expected damage to policy makers.

The 2008 financial crisis appears, after a shallow investigation, to be sufficiently analogous to AGI risk that it should give us some small reason to be concerned that policy-makers will not manage the invention of AGI wisely.

The risks to critical infrastructure from geomagnetic storms are far too small to be in the same reference class with risks from AGI.

The eradication of smallpox is only somewhat analogous to the invention of AGI.

Jonah performed very shallow investigations of how policy-makers have handled risks from cyberwarfare, chlorofluorocarbons, and the Cuban m

...
A concise version of “Twelve Virtues of Rationality”, with Anki deck
by  — 2013-09-12T02:38:45.911Z

In an effort to internalise the Twelve Virtues of Rationality, I created an Anki deck. It's already been done, so the reason I'm posting is to share a condensed version of the article (created as a side effect of my making the deck).

Hopefully it will make it easier to quickly refresh the concepts if you've already read the article.

If you're not using spaced repetition, you should. Don't believe me? Try reading Gwern's thorough review of the topic.

Then download the “Twelve Virtues of Rationality” deck.

The twelve virtues of rationality: Curiosity, relinquishment, lightness, evenness, argument, empiricism, simplicity, humility, perfectionism, precision, scholarship, and the void.

The first virtue is curiosity.

Curiosity seeks to annihilate itself; there is no curiosity that does not want an answer.

A burning itch to know is higher than a solemn vow to pursue truth.

To feel the burning itch of curiosity requires both that you be ignorant, and that you desire to relinquish your ignorance.

The second virtue is relinquishment.

P. C. Hodgell said: “That which can be destroyed by the truth should be.”

If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear.

Evaluate your beliefs first and then arrive at your emotions.

The third virtue is lightness.

Surrender to the truth as quickly as you can.

If you regard evidence as a constraint and seek to free yourself, you sell yourself into the chains of your whims.

The fourth virtue is evenness.

Beware lest you place huge burdens of proof only on propositions you dislike, and then defend yourself by saying: “But it is good to be skeptical.”

Do not seek to argue for one side or another, for if you knew your destination, you would already be there.

To be clever in argument is not rationality but rationalization.

The fifth virtue is argument.

Those who smile wisely and say: “I will not argue” remove themselves from help, and withdraw from the communal effort.

The part of yourself that distorts what you say to others also distorts your own thoughts.

Seek a test that lets reality judge between you.

The sixth virtue is empiricism.

The roots of knowledge are in observation and its fruit is prediction.

Do not ask which beliefs to profess, but which experiences to anticipate.

The seventh virtue is simplicity.

...
Meetup : Helsinki Meetup
by mikko — 2013-09-11T17:21:42.973Z

Discussion article for the meetup : Helsinki Meetup

WHEN:
22 September 2013 02:00:00PM (+0300)

WHERE:
Helsinki Music Centre, 00100 Helsinki, Finland

The theme of this meetup will be strategic thinking, with Kaj Sotala leading the discussion.

We'll meet in the cafe at Helsinki Music Centre. Afterwards we'll probably move to Kaisla for socializing.

Discussion article for the meetup : Helsinki Meetup
Definition of AI Friendliness
by  — 2013-09-11T14:55:58.368Z

How will we know if future AI’s (or even existing planners) are making decisions that are bad for humans unless we spell out what we think is unfriendly?

At a machine level the AI would be recursively minimising cost functions to produce the most effective plan of action to achieve the goal, but how will we know if its decision is going to cause harm?

Is there a model or dataset which describes what is friendly to humans? e.g.

Context

0 - running a simulation in a VM

2 - physical robot with vacuum attachment

9 - full control of a plane

Actions

0 - selecting a song to play

5 - deciding which section of floor to vacuum

99 - deciding who is an ‘enemy’

9999 - aiming a gun at an ‘enemy’

Impact

1 - poor song selected to play, human mildly annoyed

2 - ineffective use of resources (vacuuming the same floor section twice)

99 - killing a human

99999 - killing all humans

This may not be possible to get agreement from all countries/cultures/beliefs, but it is something we should discuss and attempt to get some agreement.

.
Meetup : Durham NC/Triangle Area: Cognitive Biases, Continued
by therufs — 2013-09-10T23:26:48.518Z

Discussion article for the meetup : Durham NC/Triangle Area: Cognitive Biases, Continued

WHEN:
12 September 2013 07:00:00PM (-0400)

WHERE:
420 West Geer St., Durham NC 27701

Follow up discussion to the prior meetup's cognitive biases survey! We will focus on higher-ranked biases from that meetup.

7:00 obtain coffees

7:30 discussion

9:30ish adjourn to Fullsteam

Discussion article for the meetup : Durham NC/Triangle Area: Cognitive Biases, Continued
The Interrupted Ultimate Newcomb's Problem
by linkhyrule5 — 2013-09-10T23:04:54.042Z

While figuring out my error in my solution to the Ultimate Newcomb's Problem, I ran across this (distinct) reformulation that helped me distinguish between what I was doing and what the problem was actually asking.

... but that being said, I'm not sure if my answer to the reformulation is correct either.

The question, cleaned for Discussion, looks like this:

You approach the boxes and lottery, which are exactly as in the UNP. Before reaching it, you come to sign with a flashing red light. The sign reads: "INDEPENDENT SCENARIO BEGIN."

Omega, who has predicted that you will be confused, shows up to explain: "This is considered an artificially independent experiment. Your algorithm for solving this problem will not be used in my simulations of your algorithm for my various other problems. In other words, you are allowed to two-box here but one-box Newcomb's problem, or vice versa."

This is motivated by the realization that I've been making the same mistake as in the original Newcomb's Problem, though this justification does not (I believe) apply to the original. The mistake is simply this: that I assumed that I simply appear in medias res. When solving the UNP, it is (seems to be) important to remember that you may be in some very rare edge case of the main problem, and that you are choosing your algorithm for the problem as a whole.

But if that's not true - if you're allowed to appear in the middle of the problem, and no counterfactual-yous are at risk - it sure seems like two-boxing is justified - as khafra put it, "trying to ambiently control basic arithmetic".

(Speaking of which, is there a write up of ambient decision theory anywhere? For that matter, is there any compilation of decision theories?)

EDIT: (Yes to the first, though not under that name: Controlling Constant Programs.)
Two special CFAR classes in Sept: Installing Habits and Rationality for Programmers
by palladias — 2013-09-10T20:29:53.519Z

The next four-day CFAR workshops are in October (Bay Area) and November (NYC), but there are two upcoming one-day classes that may be of interest to LessWrongers.

Making Habits that Stick - Sept 15, 3hrs, $90

Learn how to make new, useful habits and replace pernicious old ones. Andrew Critch, one of CFAR’s instructors will be teaching you how the brain holds on to patterns and cached responses. And then you’ll make use of your brain’s circuitry to set new habits deliberately.

Alumni of our four-day workshops have used this material to do anything from remembering to check a to do list, right when you walk through the door at home, to making sure that a feeling of confusion triggers you to speak up and ask for an example.

One-Day Workshop for Programmers - Sept 29, full day, $330

Take a selection of classes from our four-day workshop that have been tailored for relevance to people in programming careers. Why? Programmers continually manage huge amounts of their own time on large projects, and they know that anything you do continually is worth setting aside time to optimize.

After spending a day with CFAR, you can expect to catch yourself making a lot more mistakes. They won’t be new; you’ll just be more attuned to how your brain’s innate heuristics can lead you astray, so you’ll catch on to slip-ups faster. And instead of feeling bad about these failure modes, you’ll have the tools you need to jump out of them quickly. In fact, it won’t feel as much like making mistakes as discovering bugs — you’ll be curious and confident about figuring out a solution, and proud of yourself for catching the opportunity.

If you're interested in our curriculum, but would like to do something smaller than a workshop because of time or money constraints, these two September offerings may be of interest.
Three ways CFAR has changed my view of rationality
by Julia_Galef — 2013-09-10T18:24:21.981Z

The Center for Applied Rationality's perspective on rationality is quite similar to Less Wrong's. In particular, we share many of Less Wrong's differences from what's sometimes called "traditional" rationality, such as Less Wrong's inclusion of Bayesian probability theory and the science on heuristics and biases.

But after spending the last year and a half with CFAR as we've developed, tested, and attempted to teach hundreds of different versions of rationality techniques, I've noticed that my picture of what rationality looks like has shifted somewhat from what I perceive to be the most common picture of rationality on Less Wrong. Here are three ways I think CFAR has come to see the landscape of rationality differently than Less Wrong typically does – not disagreements per se, but differences in focus or approach. (Disclaimer: I'm not speaking for the rest of CFAR here; these are my own impressions.)

1. We think less in terms of epistemic versus instrumental rationality.

Formally, the methods of normative epistemic versus instrumental rationality are distinct: Bayesian inference and expected utility maximization. But methods like "use Bayes' Theorem" or "maximize expected utility" are usually too abstract and high-level to be helpful for a human being trying to take manageable steps towards improving her rationality. And when you zoom in from that high-level description of rationality down to the more concrete level of "What five-second mental habits should I be training?" the distinction between epistemic and instrumental rationality becomes less helpful.

Here's an analogy: epistemic rationality is like physics, where the goal is to figure out what's true about the world, and instrumental rationality is like engineering, where the goal is to accomplish something you want as efficiently and effectively as possible. You need physics to do engineering; or I suppose you could say that doing engineering is doing physics, but with a practical goal. However, there's plenty of physics that's done for its own sake, and doesn't have obvious practical applications, at least not yet. (String theory, for example.) Similarly, you need a fair amount of epistemic rationality in order to be instrumentally rational, though there are parts of epistemic rationality that many of us practice for their own sake, and not as a means to an end. (For example, I apprecia...
The Ultimate Newcomb's Problem
by Eliezer Yudkowsky — 2013-09-10T02:03:25.897Z

You see two boxes and you can either take both boxes, or take only box B. Box A is transparent and contains $1000. Box B contains a visible number, say 1033. The Bank of Omega, which operates by very clear and transparent mechanisms, will pay you $1M if this number is prime, and $0 if it is composite. Omega is known to select prime numbers for Box B whenever Omega predicts that you will take only Box B; and conversely select composite numbers if Omega predicts that you will take both boxes. Omega has previously predicted correctly in 99.9% of cases.

Separately, the Numerical Lottery has randomly selected 1033 and is displaying this number on a screen nearby. The Lottery Bank, likewise operating by a clear known mechanism, will pay you $2 million if it has selected a composite number, and otherwise pay you $0. (This event will take place regardless of whether you take only B or both boxes, and both the Bank of Omega and the Lottery Bank will carry out their payment processes - you don't have to choose one game or the other.)

You previously played the game with Omega and the Numerical Lottery a few thousand times before you ran across this case where Omega's number and the Lottery number were the same, so this event is not suspicious.

Omega also knew the Lottery number before you saw it, and while making its prediction, and Omega likewise predicts correctly in 99.9% of the cases where the Lottery number happens to match Omega's number. (Omega's number is chosen independently of the lottery number, however.)

You have two minutes to make a decision, you don't have a calculator, and if you try to factor the number you will be run over by the trolley from the Ultimate Trolley Problem.

Do you take only box B, or both boxes?
Military Rationalities and Irrationalities
by pscheyer — 2013-09-09T23:48:09.525Z

In response to the question

"Does anyone happen to know of reliable ways for increasing one's supply of executive function, by the way? I seem to run out of it very quickly in general."

(Kaj_Solata)

I posted that my military experience seems effectively designed to increase executive function. Some examples of this from myself and metastable are

Uniforms- not having to think about your wardrobe, ever, saves a lot of time, mental effort, and money. Steve Jobs and President Obama are known for also using uniforms specifically for this purpose.

PT- Daily, routinized exercise. Done in a way that very few people are deciding what comes next.

-Maximum use of daylight hours

Med Group and Force Support-Minimized high-risk projects outside of workplace (paternalistic health care, insurance, and in many cases, housing and continuing education.)

After a moment's thought it occurred to me that there are some double-edged swords in Military Rationality as well, some of which lead to classic jokes like 'Military Intelligence is an oxymoron.'

Regulations- A select few 'experts' create policies which everyone else is required to follow at all times. Unfortunately these experts are never (never ever) encouraged to consider knock-on effects. Ugh.

Anybody else have insights on the military they want to share here? I feel a couple of good posts on increasing executive function might come out of a discussion on the rationalities and irrationalities of the armed forces.
The State of the Art of Scientific Research on Polyamoury
by Ritalin — 2013-09-09T21:26:42.463Z

The idea of polyamoury is one that interests me. However, while such books as The Ethical Slut have done a good job of providing me with tools to understand and possibly handle the challenges and rewards involved, I found them unsatisfying in that they were largely based on anecdotal evidence, with a very strong selection bias. Before making the jump of attempting to live that way, one would need to know precisely the state of the art of scientific, rigourous, credible research on the topic; it is a tedious job to seek out and compile everything, but I believe it is a job worth doing.

I'll be initiating an ongoing process of data compilation, and will publish my findings on this thread as I discover and summarize them. Any help is greatly appreciated, as this promises to be long and tedious. I might especially need help extracting meaningful information from the masses of data; I am not a good statistician yet, far from it.

To Be Expanded...
Satisficing versus optimizing in instructor selection
by VipulNaik — 2013-09-09T19:43:33.569Z

In my previous post, I argued both that instructor selection is relatively neglected in undergraduate courses, and that criteria that students routinely use for instructor selection may be suboptimal. I outlined some possible pointers regarding good instructor selection, based on my own understanding of how the process works. There were a large number of thoughtful responses to my post. The most interesting responses argued that people do, and/or should, satisfice with respect to instructor selection.

I raised the point that people tend to satisfice rather than maximize in one of my own comments on the post:

So, to clarify, although I think it "often" happens that people select instructors randomly, it doesn't always happen. My impression is that there's heavy availability bias: if students can easily access evaluations, or know friends who have taken classes with a particular instructor, they'll use that data. If not, they'll select randomly. In general, my impression is that students are risk-averse and satisfice on this count: as long as an instructor seems "good enough" they won't generally try to look for a better one (this is particularly true for people in multi-course sequences with an instructor, we could call this a "status quo bias" or an "endowment effect" depending on your perspective). More proactive approaches, such as sitting in on classes that the instructors are teaching in the previous term, seem to be relatively rare.

My other point is that even when students are making proactive choices, the criteria they use may be suboptimal. I am not aware of high quality advice that would help students select instructors effectively. This is, I believe, in contrast with the vast (though possibly not very high quality) literature available on how to select a college or a major. If intra-institutional variation in instructor quality is comparable to inter-institutional variation, then there are probably unrealized gains in selecting instructors according to better criteria.

In response, Linkhyrule5 wrote:

One thing I would note is that good teachers are rare and not very "good," but bad teachers are common and very very bad. This ties into Trevor's post: once you've avoided the bad teachers, it's probably not worth the much larger effort required to find an extra-bonus-good teacher that may or may not exist.

and elaborated in the same thread:

I shou

...