trentmkelly/LessWrong-43k huggingface dataset:
[LINK] Counterfactual Strategies
by Strilanc — 2014-06-17T19:29:30.438Z

Link:

Counterintuitive Counterfactual Strategies

Overview:

Over the weekend, I was thinking about the variant of Newcomb's Paradox where both boxes are transparent. The one where, unless you precommit to taking a visibly empty box instead of both boxes, omega can self-consistently give you less money.

I was wondering if I could make this kind of "sacrifice yourself for yourself" situation happen without involving a predictor guessing your choice before you made it. Turns out you can.
An extended class of utility functions
by Stuart_Armstrong — 2014-06-17T16:36:26.714Z

This is a technical result that I wanted to check before writing up a major piece on value loading.

The purpose of a utility function is to give an agent criteria with which to make a decision. If two utility functions always give the same decisions, they're generally considered the same utility function. So, for instance, the utility function u always gives the same decisions as u+C for some constant C, or Du for some positive constant D. Thus we can say that utility functions are equivalent if they are related by a positive affine transformation.

For specific utility functions, and specific agents, the class of functions that give the same decisions is quite a bit larger. For instance, imagine that v is a utility function with the property v("any universe which contains humans")=constant. Then any human who attempts to follow u, could equivalently follow u+v (neglecting acausal trade) - it makes no difference. In general, if no action the agent could ever take would change the value of v, then u and u+v give the same decisions.

More subtly, if the agent can change v but cannot change the expectation of v, then u and u+v still give the same decisions. This is because for any actions a and b the agent could take:

E(u+v | a) = E(u | a) + E(v | a) = E(u | a) + E(v | b).

Hence E(u+v | a) > E(u+v | b) if and only if E(u | a) > E(u | b), and so the decision hasn't changed.

Note that E(v | a) need not be constant for all actions: simply that for every actions and b that an agent could take at a particular decision point, E(v | a) = E(v | b). It's perfectly possible for the expectation of v to be different at different moments, or conditional on different decisions made at different times.

Finally, as long as v obeys the above properties, there is no reason for it to be a utility function in the classical sense - it could be constructed any way we want.

An example: suffer not from probability, nor benefit from it

The preceding seems rather abstract, but here is the motivating example. It's a correction term T that adds or subtracts utility, as external evidence comes in (it's important that the evidence is external - the agent gets no correction from knowing what its own actions are/were). If the AI knows evidence e, and new (external) evidence f comes in, then its utility gets adjusted by T(e,f) which is defined as

T(e,f) = E(u | e) - E(u | e, f)

In othe...
Value learning: ultra-sophisticated Cake or Death
by Stuart_Armstrong — 2014-06-17T16:36:12.409Z

Many mooted AI designs rely on "value loading", the update of the AI’s preference function according to evidence it receives. This allows the AI to learn "moral facts" by, for instance, interacting with people in conversation ("this human also thinks that death is bad and cakes are good – I'm starting to notice a pattern here"). The AI has an interim morality system, which it will seek to act on while updating its morality in whatever way it has been programmed to do.

But there is a problem with this system: the AI already has preferences. It is therefore motivated to update its morality system in a way compatible with its current preferences. If the AI is powerful (or potentially powerful) there are many ways it can do this. It could ask selective questions to get the results it wants (see this example). It could ask or refrain from asking about key issues. In extreme cases, it could break out to seize control of the system, threatening or imitating humans so it could give itself the answers it desired.

Avoiding this problem turned out to be tricky. The Cake or Death post demonstrated some of the requirements. If p(C(u)) denotes the probability that utility function u is correct, then the system would update properly if:

Expectation(p(C(u)) | a) = p(C(u)).

Put simply, this means that the AI cannot take any action that could predictably change its expectation of the correctness of u. This is an analogue of the conservation of expected evidence in classical Bayesian updating. If the AI was 50% convinced about u, then it could certainly ask a question that would resolve its doubts, and put p(C(u)) at 100% or 0%. But only as long as it didn't know which moral outcome was more likely.

That formulation gives too much weight to the default action, though. Inaction is also an action, so a more correct formulation would be that for all actions a and b,

Expectation(p(C(u)) | a) = Expectation(p(C(u)) | b).

How would this work in practice? Well, suppose an AI was uncertain between whether cake or death was the proper thing, but it knew that if it took action a:"Ask a human", the human would answer "cake", and it would then update its values to reflect that cake was valuable but death wasn't. However, the above condition means that if the AI instead chose the action b:"don't ask", exactly the same thing would happen.

In practice, this mean...
[LINK] The errors, insights and lessons of famous AI predictions: preprint
by Stuart_Armstrong — 2014-06-17T14:32:27.502Z

A preprint of the "The errors, insights and lessons of famous AI predictions – and what they mean for the future" is now available on the FHI's website.

Abstract:

Predicting the development of artificial intelligence (AI) is a difficult project – but a vital one, according to some analysts. AI predictions are already abound: but are they reliable? This paper starts by proposing a decomposition schema for classifying them. Then it constructs a variety of theoretical tools for analysing, judging and improving them. These tools are demonstrated by careful analysis of five famous AI predictions: the initial Dartmouth conference, Dreyfus's criticism of AI, Searle's Chinese room paper, Kurzweil's predictions in the Age of Spiritual Machines, and Omohundro's ‘AI drives’ paper. These case studies illustrate several important principles, such as the general overconfidence of experts, the superiority of models over expert judgement and the need for greater uncertainty in all types of predictions. The general reliability of expert judgement in AI timeline predictions is shown to be poor, a result that fits in with previous studies of expert competence.

The paper was written by me (Stuart Armstrong), Kaj Sotala and Seán S. Ó hÉigeartaigh, and is similar to the series of Less Wrong posts starting here and here.
Meetup : Canberra: Many Worlds + Paranoid Debating
by DanielFilan — 2014-06-17T13:44:22.361Z

Discussion article for the meetup : Canberra: Many Worlds + Paranoid Debating

WHEN:
27 June 2014 06:00:00PM (+1000)

WHERE:
108 North Road, Acton, ACT

First, I will give a brief presentation about the many worlds interpretation of quantum mechanics, talking about what it is and how it attempts to deal with issues surrounding probability.

Next, we will have another game of paranoid debating. This time, so that the 'mole' knows the answer to the question, we will ask that everyone come up with two questions before the event that they know the answer to - one place to find questions would be digging through Wikipedia's list of lists http://en.wikipedia.org/wiki/List_of_lists - and write them down, putting all the questions into the hat. If your question comes up, you will be the mole.

As always, vegan snacks will be provided.

General meetup info:

If you use Facebook, please join our group: https://www.facebook.com/groups/lwcanberra/

Structured meetups are held on the second Saturday and fourth Friday of each month from 6 pm until late in the CSIT building, room N101.

There will be LWers at the Computer Science Students Association's weekly board games night, held on Wednesdays from 7 pm in the same location.

Discussion article for the meetup : Canberra: Many Worlds + Paranoid Debating
Thoughts on Structuring AI Thought
by  — 2014-06-17T11:33:57.415Z

Introduction

(post has been edited on 17.6 18:50 GMT, there's is an added part)

I'd like to write my thoughts about structuring the psyche for an AI. However I am not an expert on computers, programming or AIs, nor even psychology or the human mind in general. In fact I have very little reason to think it should be me writing about this topic. I think that's ok. It's a thread to share thoughts and interact. Part of the problem is socially managing this message. It requires a certain degree of boldness to present your thoughts. It think also requires a certain degree of uncertainty to improve those thoughts.

The content of this thread will be on a very general and vague level. I would be more specific, if I could. But it's hard enough as it is. Perhaps this thread will stimulate thoughts for those who are more adept with the matter. Most of how I think about this, originates from my personal theory of mind, which probably is not anything special, and therefore it resembles an attempt of emulating the human brain. This according to some posts written on lesswrong would be a socially approved signal for bad thinking perhaps.

None of this probably makes sense. And it's probably too general to have any real meaning. But I hope this post is useful to stimulate thought if nothing else. I'm a little worried about potential criticism, since I kind of feel like I'm trying to take a too large bit of a cake that's really not meant for me.. But that's ok.

Let's move on.

Concept of time

I think understanding the concept of time essentially requires arranging things into a series. Objects in the series can have an order in the sense that some objects come before, others come later. Based on this very simple sequential thought can be constructed a sense for passage of time by adding the notion that the present time can be some object on such series.

Logical sequences as part of sequential reasoning

If you can model sequences with an idea of something that is now and later or before, it allows formulating some sort of logic with the idea of moving towards some state in that sequence.

Decision making by altering the states of sequences

So understanding decisions requires the idea of influencing an outcome that resides somewhere along the way of a sequence.

Goals and utility functions

Goals are intents of changing something that comes later in the sequence. Then wha...
[LINK] The future of the Turing test and intelligence measures
by Stuart_Armstrong — 2014-06-17T11:12:13.307Z

Following the recent hype over the potential of a machine passing of the Turing test, Adam Ford interviews Stuart Armstrong (me) of the FHI about the meaning of the test, how we can expect a future of many upcoming "Turing test passings" according to varying criteria of strictness, and how and why we test for intelligence in the first place.

I predict that we are entering an era where "X passed the Turing test" will be a more and more common announcement, followed by long discussions as to whether that was a true pass or not.
Maximize Worst Case Bayes Score
by Scott Garrabrant — 2014-06-17T09:12:14.367Z

In this post, I propose an answer to the following question:

Given a consistent but incomplete theory, how should one choose a random model of that theory?

My proposal is rather simple. Just assign probabilities to sentences in such that if an adversary were to choose a model, your Worst Case Bayes Score is maximized. This assignment of probabilities represents a probability distribution on models, and choose randomly from this distribution. However, it will take some work to show that what I just described even makes sense. We need to show that Worst Case Bayes Score can be maximized, that such a maximum is unique, and that this assignment of probabilities to sentences represents an actual probability distribution. This post gives the necessary definitions, and proves these three facts.

Finally, I will show that any given probability assignment is coherent if and only if it is impossible to change the probability assignment in a way that simultaneously improves the Bayes Score by an amount bounded away from 0 in all models. This is nice because it gives us a measure of how far a probability assignment is from being coherent. Namely, we can define the "incoherence" of a probability assignment to be the supremum amount by which you can simultaneously improve the Bayes Score in all models. This could be a useful notion since we usually cannot compute a coherent probability assignment so in practice we need to work with incoherent probability assignments which approach a coherent one.

I wrote up all the definitions and proofs on my blog, and I do not want to go through the work of translating all of the latex code over here, so you will have to read the rest of the post there. Sorry. In case you do not care enough about this to read the formal definitions, let me just say that my definition of the "Bayes Score" of a probability assignment P with respect to a model M is the sum over all true sentences s of m(s)log(P(s)) plus the sum over all false sentences s of m(s)log(1-P(s)), where m is some fixed nowhere zero probability measure on all sentences. (e.g. m(s) is 1/2 to the number of bits needed to encode s)

I would be very grateful if anyone can come up with a proof that this probability distribution which maximizes Worst Case Bayes Score has the property that its Bayes Score is independent of the choice of what model we use to judge it. I believe i...
From "Coulda" and "Woulda" to "Shoulda": Predicting Decisions to Minimize Regret for Partially Rational Agents
by  — 2014-06-16T21:02:18.406Z

TRIGGER WARNING: PHILOSOPHY. All those who believe in truly rigorous, scientifically-grounded reasoning should RUN AWAY VERY QUICKLY.

Abstract: Human beings want to make rational decisions, but their decision-making processes are often inefficient, and they don't possess direct knowledge of anything we could call their utility functions. Since it is much easier to detect a bad world state than a good one (there are vastly more of them, so less information is needed to classify accurately), humans tend to have an easy time detecting bad states, but this emotional regret is no more useful for formal reasoning about human rationality, since we don't possess a causal model of it in terms of decision histories and outcomes. We tackle this problem head-on, assuming only that humans can reason over a set of beliefs and a perceived state of the world to generate a probability distribution over actions.

Consider rationality: optimizing the world to better and better match a utility function, which is itself complete, transitive, continuous, and gives results which are independent of irrelevant alternatives. Now consider actually existing human beings: creatures who can often and easily be tricked into taking Dutch Book bets through exploitation of their cognitive structure, without even having to go to the trouble of actually deceiving them with regards to specific information.

Consider that being one of those poor sods must totally suck. We believe this provides sufficient motivation for wanting to help them out a bit. Unfortunately, doing so is not very simple: since they didn't evolve as rational creatures, it's very easy to propose an alternate set of values that captures absolutely nothing of what they actually want out of life. In fact, since they didn't even evolve as 100% self-aware creatures, their emotional qualia are not even reliable indicators of anything we would call a proper utility function. They know there's something they want out of life, and they know they don't know what it is, but that doesn't help because they still don't know what it is, and knowledge of ignorance does not magically reduce the ignorance.

So! How can we help them without just overriding them or enslaving them to strange and alien cares? Well, one barest rudiment of rationality with which evolution did manage to bless them...
Group Rationality Diary, June 16-30
by therufs — 2014-06-16T17:54:29.282Z

This is the public group instrumental rationality diary for June 16-30.

It's a place to record and chat about it if you have done, or are actively doing, things like:

Established a useful new habit

Obtained new evidence that made you change your mind about some belief

Decided to behave in a different way in some set of situations

Optimized some part of a common routine or cached behavior

Consciously changed your emotions or affect with respect to something

Consciously pursued new valuable information about something that could make a big difference in your life

Learned something new about your beliefs, behavior, or life that surprised you

Tried doing any of the above and failed

Or anything else interesting which you want to share, so that other people can think about it, and perhaps be inspired to take action themselves. Try to include enough details so that everyone can use each other's experiences to learn about what tends to work out, and what doesn't tend to work out.

Thanks to cata for starting the Group Rationality Diary posts, and to commenters for participating.

Previous diary: June 1-15

Next diary: July 1-15

Rationality diaries archive
The Power of Noise
by jsteinhardt — 2014-06-16T17:26:30.329Z

Recently Luke Muelhauser posed the question, “Can noise have power?”, which basically asks whether randomization can ever be useful, or whether for every randomized algorithm there is a better deterministic algorithm. This question was posed in response to a debate between Eliezer Yudkowsky and Scott Aaronson, in which Eliezer contends that randomness (or, as he calls it, noise) can't ever be helpful, and Scott takes the opposing view. My goal in this essay is to present my own views on this question, as I feel many of them have not yet been brought up in discussion.

I'll spare the reader some suspense and say that I basically agree with Scott. I also don't think – as some others have suggested – that this debate can be chalked up to a dispute about the meaning of words. I really do think that Scott is getting at something important in the points he makes, which may be underappreciated by those without a background in a field such as learning theory or game theory.

Before I start, I'd like to point out that this is really a debate about Bayesianism in disguise. Suppose that you're a Bayesian, and you have a posterior distribution over the world, and a utility function, and you are contemplating two actions A and B, with expected utilities U(A) and U(B). Then randomly picking between A and B will have expected utility , and so in particular at least one of A and B must have higher expected utility than randomizing between them. One can extend this argument to show that, for a Bayesian, the best strategy is always deterministic. Scott in fact acknowledges this point, although in slightly different language:

“Randomness provably never helps in average-case complexity (i.e., where you fix the probability distribution over inputs) -- since given any ensemble of strategies, by convexity there must be at least one deterministic strategy in the ensemble that does at least as well as the average.” -Scott Aaronson

I think this point is pretty iron-clad and I certainly don't wish to argue against it. Instead, I'd like to present several examples of scenarios where I will argue that randomness clearly is useful and necessary, and use this to argue that, at least in these scenarios, one should abandon a fully Bayesian stance. At the meta level, this essay is therefore an argument in favor of maintaining multiple paradigms (in the Kuhnian s

...
Open thread, 16-22 June 2014
by David_Gerard — 2014-06-16T13:12:45.789Z

Previous open thread

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one.

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Meetup : Atlanta Movie Night: Introduction to Machine Learning
by Adele_L — 2014-06-15T20:42:09.239Z

Discussion article for the meetup : Atlanta Movie Night: Introduction to Machine Learning

WHEN:
22 June 2014 07:00:35PM (-0400)

WHERE:
2388 Lawrenceville Hwy. Unit L Decatur, GA 30033

Stanford professor, Andrew Ng, is teaching a course on Coursera about machine learning (ML). Even if you never write an ML program, the videos of the course provide a nice overview of the applications of ML and the general techniques used. As such, come join us for a movie watching party: we’ll eat popcorn and watch the first’s week’s videos (about 40 minutes), and then discuss what we’ve learned and it’s implications for the creation of truly thinking machines.

See y’all!

As usual, add any topics you’d like to present on or hear about to the following document:
https://docs.google.com/a/nnadi.org/document/d/1vYFn08DrveXON0l5tiMzACHzVsf8_Mwfa2ZYfuMzQQQ/edit

Discussion article for the meetup : Atlanta Movie Night: Introduction to Machine Learning
Some alternatives to “Friendly AI”
by lukeprog — 2014-06-15T19:53:20.340Z

Cross-posted from my blog.

What does MIRI's research program study?

The most established term for this was coined by MIRI founder Eliezer Yudkowsky: "Friendly AI." The term has some advantages, but it might suggest that MIRI is trying to build C-3PO, and it sounds a bit whimsical for a serious research program.

What about safe AGI or AGI safety? These terms are probably easier to interpret than Friendly AI. Also, people like being safe, and governments like saying they're funding initiatives to keep the public safe.

A friend of mine worries that these terms could provoke a defensive response (in AI researchers) of "Oh, so you think me and everybody else in AI is working on unsafe AI?" But I've never actually heard that response to "AGI safety" in the wild, and AI safety researchers regularly discuss "software system safety" and "AI safety" and "agent safety" and more specific topics like "safe reinforcement learning" without provoking negative reactions from people doing regular AI research.

I'm more worried that a term like "safe AGI" could provoke a response of "So you're trying to make sure that a system which is smarter than humans, and able to operate in arbitrary real-world environments, and able to invent new technologies to achieve its goals, will be safe? Let me save you some time and tell you right now that's impossible. Your research program is a pipe dream."

My reply goes something like "Yeah, it's way beyond our current capabilities, but lots of things that once looked impossible are now feasible because people worked really hard on them for a long time, and we don't think we can get the whole world to promise never to build AGI just because it's hard to make safe, so we're going to give AGI safety a solid try for a few decades and see what can be discovered." But that's probably not all that reassuring.

How about high-assurance AGI? In computer science, a "high assurance system" is one built from the ground up for unusually strong safety and/or security guarantees, because it's going to be used in safety-critical applications where human lives — or sometimes simply billions of dollars — are at stake (e.g. autopilot software or Mars rover software). So there's a nice analogy to MIRI's work, where w...