trentmkelly/LessWrong-43k huggingface dataset:
Can science come to understand consciousness? A problem of philosophical zombies (Yes, I know, P-zombies again.)
by Capla — 2014-11-17T17:06:24.798Z

In response to the classic Mysterious Answers to Mysterious Questions, I express some skepticism that consciousness is can be understood by science. I postulate (with low confidence) that consciousness is “inherently mysterious”, in that it is philosophically and scientifically impenetrable. The mysteriousness is a fact about our state of mind, but that state of mind is due to a fundamental epistemic feature of consciousness and is impossible to resolve.

My issue with understanding the cause of consciousness involves p-zombies. Any experiment with the goal of understanding consciousness would have to be able to detect consciousness, which seems to me to be philosophically impossible. To be more specific, any scientific investigation of the cause of consciousness would have (to simplify) an independent variable that we could manipulate to see how this manipulation affects the dependent variable, the presence or absence of consciousness. We assume that those around us are conscious, and we have good reason to do so, but we can't rely on that assumption in any experiment in which we are investigating consciousness. Before we ask “what is causing x?”, we first have to know that x is present.

As Eliezer points out, that an individual says he's conscious is a pretty good signal of consciousness, but we can't necessarily rely on that signal for non-human minds. A conscious AI may never talk about its internal states depending on its structure. (Humans have a survival advantage to social sharing of internal realities; an AI will not be subject to that selection pressure. There’s no reason for it to have any sort of emotional need to share its feelings, for example.) On the flip side, a savvy but non-conscious AI may talk about it's "internal states", not because it actually has internal states, but because it is “guessing the teacher's password” in the strongest way imaginable: it has no understanding whatsoever of what those states are, but computes that aping internal states will accomplish it's goals. I don't know how we could possibly know if the AI is aping consciousness for it own ends or if it actually is conscious. If consciousness is thus undetectable, I can't see how science can investigate it.

That said, I am very well aware that “Throughout history, every mystery, ever solved has turned out to b...
Systemic risk: a moral tale of ten insurance companies
by Stuart_Armstrong — 2014-11-17T16:43:12.392Z

Once upon a time...

Imagine there were ten insurance sectors, each sector being a different large risk (or possibly the same risks, in different geographical areas). All of these risks are taken to be independent.

To simplify, we assume that all the risks follow the same yearly payout distributions. The details of the distribution doesn't matter much for the argument, but in this toy model, the payouts follow the discrete binomial distribution with n=10 and p=0.5, with millions of pounds as the unit:

This means that the probability that each sector pays out £n million each year is (0.5)10 . 10!/(n!(10-n)!).

All these companies are bound by Solvency II-like requirements, that mandate that they have to be 99.5% sure to payout all their policies in a given year - or, put another way, that they only fail to payout once in every 200 years on average. To do so, in each sector, the insurance companies have to have capital totalling £9 million available every year (the red dashed line).

Assume that each sector expects £1 million in total yearly expected profit. Then since the expected payout is £5 million, each sector will charge £6 million a year in premiums. They must thus maintain a capital reserve of £3 million each year (they get £6 million in premiums, and must maintain a total of £9 million). They thus invest £3 million to get an expected profit of £1 million - a tidy profit!

Every two hundred years, one of the insurance sectors goes bust and has to be bailed out somehow; every hundred billion trillion years, all ten insurance sectors go bust all at the same time. We assume this is too big to be bailed out, and there's a grand collapse of the whole insurance industry with knock on effects throughout the economy.

But now assume that insurance companies are allowed to invest in each other's sectors. The most efficient way of doing so is to buy equally in each of the ten sectors. The payouts across the market as a whole are now described by the discrete binomial distribution with n=100 and p=0.5:

This is a much narrower distribution (relative to its mean). In order to have enough capital to payout 99.5% of the time, the whole industry needs only keep £63 million in capital (the red dashed line). Note that this is far less that the combined capital for each sector when they were separate, which w...
I just increased my Altruistic Effectiveness and you should too
by AABoyles — 2014-11-17T15:45:00.537Z

I was looking at the marketing materials for a charity (which I'll call X) over the weekend, when I saw something odd at the bottom of their donation form:

Check here to increase your donation by 3% to defray the cost of credit card processing.

It's not news to me that credit card companies charge merchants a cut of every transaction. But the ramifications of this for charitable contributions had never sunk in. I use my credit card for all of the purchases I can (I get pretty good cash-back rates). Automatically drafting from my checking account (like a check, only without the check) costs X nothing. So I've increased the effectiveness of my charitable contributions by a small (<3%) amount by performing what amounts to a paperwork tweak.

If you use a credit card for donations, please think about making this tweak as well!
Lying in negotiations: a maximally bad problem
by Stuart_Armstrong — 2014-11-17T15:17:29.062Z

In a previous post, I showed that the Nash Bargaining Solution (NBS), the Kalai-Smorodinsky Bargaining Solution (KSBS) and own my Mutual Worth Bargaining Solution (MWBS) were all maximally vulnerable to lying. Here I can present a more general result: all bargaining solutions are maximally vulnerable to lying.

Assume that players X and Y have settled on some bargaining solution (which only cares about the defection point and the utilities of X and Y). Assume further that player Y knows player X's utility function. Let player X look at the possible outcomes, and let her label any outcome O "admissible" if there is some possible bargaining partner YO with utility function uO such that O would be the outcome of the bargain between X and YO.

For instance, in the case of NBS and KSBS, the admissible outcomes would be the outcomes Pareto-better than the disagreement point. The MWBS has a slightly larger set of admissible outcomes, as it allows players to lose utility (up to the maximum they could possibly gain).

Then the general result is:

If player Y is able to lie about his utility function while knowing player X's true utility (and player X is honest), he can freely select his preferred outcome among the outcomes that are admissible.

The proof of this is also derisively brief: player Y need simply claim to have utility uO, in order to force outcome O.

Thus, if you've agreed on a bargaining solution, all that you've done is determined the set of outcomes among which your lying opponent will freely choose.

There may be a subtlety: your system could make use of an objective (or partially objective) disagreement point, which your opponent is powerless to change. This doesn't change the result much:

If player Y is able to lie about his utility function while knowing player X's true utility (and player X is honest), he can freely select his preferred outcome among the outcomes that are admissible given the disagreement point.

Exploitation and gains from trade

Note that the above result did not make any assumptions about the outcome being Pareto - giving up Pareto doesn't make you non-exploitable (or "strategyproof" as it is often called).

But note also that the result does not mean that the system is exploitable! In the random dictator setup, you randomly assign power to one player, who then makes all the decisions. In terms of expected utilit...
Open thread, Nov. 17 - Nov. 23, 2014
by MrMind — 2014-11-17T08:25:58.177Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Musk on AGI Timeframes
by Artaxerxes — 2014-11-17T01:36:12.012Z

Elon Musk submitted a comment to edge.org a day or so ago, on this article. It was later removed.

The pace of progress in artificial intelligence (I'm not referring to narrow AI) is incredibly fast. Unless you have direct exposure to groups like Deepmind, you have no idea how fast-it is growing at a pace close to exponential. The risk of something seriously dangerous happening is in the five year timeframe. 10 years at most. This is not a case of crying wolf about something I don't understand.

I am not alone in thinking we should be worried. The leading AI companies have taken great steps to ensure safety. The recognize the danger, but believe that they can shape and control the digital superintelligences and prevent bad ones from escaping into the Internet. That remains to be seen...

Now Elon has been making noises about AI safety lately in general, including for example mentioning Bostrom's Superintelligence on twitter. But this is the first time that I know of that he's come up with his own predictions of the timeframes involved, and I think his are rather quite soon compared to most.

The risk of something seriously dangerous happening is in the five year timeframe. 10 years at most.

We can compare this to MIRI's post in May this year, When Will AI Be Created, which illustrates that it seems reasonable to think of AI as being further away, but also that there is a lot of uncertainty on the issue.

Of course, "something seriously dangerous" might not refer to full blown superintelligent uFAI - there's plenty of space for disasters of magnitude in between the range of the 2010 flash crash and clippy turning the universe into paperclips to occur.

In any case, it's true that Musk has more "direct exposure" to those on the frontier of AGI research than your average person, and it's also true that he has an audience, so I think there is some interest to be found in his comments here.
Approximability
by abramdemski — 2014-11-17T00:41:37.000Z

I've maintained a consistent focus on the approximable when it comes to idealized theories of intelligence. I don't care much how long it takes to converge, but if there \emph{is} no convergent approximation, I figure all bets are off. This is somewhat nonstandard.

There are many levels at which we could focus attention. The real target for algorithms is, of course, practical utility. This has several proxies, including polynomial-time computability and polynomial-time approximability (of various kinds).

Going beyond this, we could focus on computable distriutions (IE, distributions which can be approximated to any desired epsilon by a terminating algorithm); distributions which can be approximated from below, from above, or both; and my typical choice, that which has a convergent approximation at all.

It is also perfectly reasonable in some cases to go to even broader levels. For example, we might have approximable agents which are indexed by ordinals, such that higher ordinals are strictly better in some sense, but we cannot take the limit to give a single approximable super-agent better than all in the set. Rather than computations that converge to a theoretical ideal, we are now forced to choose some sufficiently high ordinal and approximate that. This kind of agent is effectively one Turing jump higher than mere approximability.

It is also possible to consider agents which encompass the whole arithmetic hierarchy, or the analytic hierarchy, or agents which have perfect knowledge of set theory, and so on.

Each of these levels gives rise to a significantly different landscape of possibilities. Our hope with any level we choose is that we can derive useful insights to apply at the practical level. However, the phenomena at each level are significantly different. A path may look promising when examined at one level and impossible when examined at another.

The example of this which I have in mind is the reflective oracle result, which intuitively tells us that it is possible for a rational Bayesian agent to understand environments which include equally-powerful agents, so long as we define a special self-reference call into the agent and allow the agents to take on mixed strategies in a way that makes consistent self-reference possible.

The method, however, is not approximable. I don't know exactly what happens when we try to put it into the realm of approximabilit

...
Meetup : MIRIxAtlanta - MIRI Research Guide + Corrigibility
by Adele_L — 2014-11-16T22:34:35.743Z

Discussion article for the meetup : MIRIxAtlanta - MIRI Research Guide + Corrigibility

WHEN:
22 November 2014 06:00:00PM (-0500)

WHERE:
2388 Lawrenceville Hwy. Unit L. Decatur, GA 30033

We'll go over the new research guide http://intelligence.org/research-guide/ which discusses which mathematical knowledge is necessary for doing FAI research, and goes over the major lines of research done at MIRI.

We will also look at a new line of research called corrigibility. From the research guide: "As artificially intelligent systems grow in intelligence and capability, some of their available options may allow them to resist intervention by their programmers. We call an AI system “corrigible” if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences."

There will also be snacks and cats! Hope to see you there!

Discussion article for the meetup : MIRIxAtlanta - MIRI Research Guide + Corrigibility
Group Rationality Diary, November 16-30
by therufs — 2014-11-16T21:16:01.043Z

This is the public group rationality diary for November 16-30.

It's a place to record and chat about it if you have done, or are actively doing, things like:

Established a useful new habit

Obtained new evidence that made you change your mind about some belief

Decided to behave in a different way in some set of situations

Optimized some part of a common routine or cached behavior

Consciously changed your emotions or affect with respect to something

Consciously pursued new valuable information about something that could make a big difference in your life

Learned something new about your beliefs, behavior, or life that surprised you

Tried doing any of the above and failed

Or anything else interesting which you want to share, so that other people can think about it, and perhaps be inspired to take action themselves. Try to include enough details so that everyone can use each other's experiences to learn about what tends to work out, and what doesn't tend to work out.

Thanks to cata for starting the Group Rationality Diary posts, and to commenters for participating.

Previous diary: November 1-15

Next diary: December 1-15

Rationality diaries archive
Agency and Life Domains
by Gleb_Tsipursky — 2014-11-16T01:38:35.777Z

Introduction

The purpose of this essay is to propose an enriched framework of thinking to help optimize the pursuit of agency, the quality of living intentionally. I posit that pursuing and gaining agency involves 3 components:

1. Evaluating reality clearly, to

2. Make effective decisions, that

3. Achieve our short and long-term goals.

In other words, agency refers to the combination of assessing reality accurately and achieving goals effectively, epistemic and instrumental rationality. The essay will first explore the concept of agency more thoroughly, and will then consider the application of this concept in different life domains, by which I mean different life areas such as work, romance, friendships, fitness, leisure, and other domains.

The concepts laid out here sprang from a collaboration between myself and Don Sutterfield, and also discussions with Max Harms, Rita Messer, Carlos Cabrera, Michael Riggs, Ben Thomas, Elissa Fleming, Agnes Vishnevkin, Jeff Dubin, and other members of the Columbus, OH, Rationality Meetup, as well as former members of this Meetup such as Jesse Galef and Erica Edelman. Members of this meetup are also collaborating to organize Intentional Insights, a new nonprofit dedicated to raising the sanity waterline through popularizing Rationality concepts in ways that create cognitive ease for a broad public audience (for more on Intentional Insights, see a fuller description here).

Agency

This section describes a framework of thinking that helps assess reality accurately and achieve goals effectively, in other words gain agency. After all, insofar as human thinking suffers from many biases, working to achieve greater agenty-ness would help us lead better lives. First, I will consider agency in relation to epistemic rationality, and then instrumental rationality: while acknowledging fully that these overlap in some ways, I believe it is helpful to handle them in distinct sections.

This essay proposes that gaining agency from the epistemic perspective involves individuals making an intentional evaluation of their environment and situation, in the moment and more broadly in life, sufficient to understand the full extent of one’s options within it and how these options relate to one’s personal short-term and long-term goals. People often make their decisions, both in the moment and major life decisions, based on socia

...
Meetup : Toronto: making hard decisions
by Giles — 2014-11-15T23:41:59.639Z

Discussion article for the meetup : Toronto: making hard decisions

WHEN:
20 November 2014 07:00:00PM (-0500)

WHERE:
Toronto

See [http://www.meetup.com/Less-Wrong-Toronto/] for the exact location.

Sometimes we are faced with a decision which is important and non-trivial. What hard decisions have you made in the past? Do you have any coming up? Do you have any techniques and strategies for making hard, real-world decisions?

For me right now a hard decision is which direction to take my career, with grad school being an option. I'd like to talk about that a bit, but other examples too, with the emphasis being on general-purpose techniques that can be applied to a wide range of decision-making problems.

Things to consider:

How do you get yourself in the right decision-making mindframe?

How do you get a handle on what your options actually are, and make sure you're considering all of them?

How do you translate your goals and values into utility, in order to compare options? Or do you not do it that way?

How do you make sure your model of the decision and its consequences squares with reality?

Discussion article for the meetup : Toronto: making hard decisions
How to build the skill and the habit of experimentation? 
by Capla — 2014-11-15T16:08:51.212Z

I want to make regular experimentation a part of my life and don't really know how. I thought that I should associate with and assist people who do run experiments (I'm interning with a psych lab and a paranormal investigator, and hope to work with some behavioral economists who run field-experiments), but I relied that I haven't taken the time to consider if that is actually a good approach or if there is somthign else I should be doing in addition.

How do I gain proficiency with experimental methods and build the habit of running simple experiments regularly? I suppose that there's a certain kind of phenomenon that to the educated mind is automatically flagged as ripe for experimentation (I'm thinking of Feynman's curiosity about the ants in his room, from Surely You're Joking, or Harry James Potter-Evans-Verres testing with his army to find out what the optimal way to fight is, prior to the first of Quirrell’s battles), but I don't have that intuition, yet.

What are the key insights, procedures, or guidelines that I need to know in order to experiment fruitfully? How do I build that intuition?

I’m looking either for recommendations or critiques. Perhaps personal experimentation is not as useful as my veneration of science in general leads me to believe? It seems to me beneficial that when faced with a problem, confusion, or dispute, one of my go-to approaches is to run an experiment.

Suggestions?
Belief Chains
by ShannonFriedman — 2014-11-15T11:09:57.217Z

A belief is an acceptance that a statement is true or that something exists. As aspiring rationalists, we strive for our beliefs to be true, accurate, and minimally biased.

You seldom see a single belief floating around. Typically beliefs tend to group into clusters and chains. In other words, if I believe that I am turning my thoughts into written words right now, that is not an isolated belief. My belief chain might look something like this:

I have sight -> The image coming into my eyes is of something that is metallic with bright lights and little boxes -> It is similar to things that have been called “computers” before -> I am wiggling my fingers to make patterns -> this is called typing -> I am typing on a computer -> the words I am thinking are being translated into writing.

Why does it matter whether I see my beliefs as chains or whether I simply look at the highest level belief such as “the words I am thinking are being translated into written word”?

It matters because at each link in the chain of belief, there is potential for falsehood to be introduced. The further I am away from the source of my high-level belief, the less likely my high-level belief is to be accurate.

Say for example that a three year old is typing on their toy computer that does not have the standard typing functionality of my computer. They could still have the same logic chain that I used:

I have sight -> The image coming into my eyes is of something that is metallic with bright lights and little boxes -> It is similar to things that have been called “computers” before -> I am wiggling my fingers to make patterns -> this is called typing -> I am typing on a computer -> the words I am thinking are being translated into writing.

Belief chains can be corrupted in many ways. Here are a few:

1. Our intuitions tell us that the more interconnecting beliefs we have, and the more agreement between different beliefs, the more likely they are to be true, right? We can check them against each other and use them as confirming evidence for one another.

These interconnections can come from the beliefs we have accumulated in our own minds, ...