trentmkelly/LessWrong-43k huggingface dataset:
My new paper: Concept learning for safe autonomous AI
by Kaj_Sotala — 2014-11-15T07:17:56.610Z

Abstract: Sophisticated autonomous AI may need to base its behavior on fuzzy concepts that cannot be rigorously defined, such as well-being or rights. Obtaining desired AI behavior requires a way to accurately specify these concepts. We review some evidence suggesting that the human brain generates its concepts using a relatively limited set of rules and mechanisms. This suggests that it might be feasible to build AI systems that use similar criteria and mechanisms for generating their own concepts, and could thus learn similar concepts as humans do. We discuss this possibility, and also consider possible complications arising from the embodied nature of human thought, possible evolutionary vestiges in cognition, the social nature of concepts, and the need to compare conceptual representations between humans and AI systems.

I just got word that this paper was accepted for the AAAI-15 Workshop on AI and Ethics: I've uploaded a preprint here. I'm hoping that this could help seed a possibly valuable new subfield of FAI research. Thanks to Steve Rayhawk for invaluable assistance while I was writing this paper: it probably wouldn't have gotten done without his feedback motivating me to work on this.

Comments welcome.
Topological truth predicates: Towards a model of perfect Bayesian agents
by Benya_Fallenstein — 2014-11-15T06:39:48.000Z

In this post, I'll introduce a new kind of self-referential "truth predicate" (of sorts), which avoids diagonalization by placing a certain topological condition on the formulas it can be applied to. In future posts, I'll show how this can be used to model perfect Bayesian agents that are able to reason about a world containing other, equally powerful agents, and how, in particular, this yields a variant of AIXI that can reason about a world containing other instances of the same kind of AIXI.

This is inspired by the way that classical game theory avoids diagonalization by its use of mixed strategies, and in fact I'll show that if they have enough common knowledge, these agents will play Nash equilibria against each other; but this framework doesn't require players to be special kinds of objects, or at least not to the degree that classical game theory does. (Besides classical game theory, Paul Christiano's reflection principle for probabilistic logic was the other main inspiration for this work.)

Perfect Bayesian agents

The fact that AIXI can't model environments containing other AIXIs isn't as specific to AIXI as it may seem; the way I see it, it's really a problem with the decision-theoretic ideal of a logically omniscient, perfect Bayesian agent. A perfect Bayesian may have uncertainty about which possible world it's living in, but given any possible world in its hypothesis space, its supposed to know everything that happens in that world (so that it can calculate expected utility). For example, if the world is some sort of deterministic cellular automaton, and our agent has uncertainty about the initial state of that automaton, then for every particular initial state, the agent needs to figure out what will happen later---despite the fact that it itself lives inside that automaton.

Of course, in reality, that isn't literally possible. But having models that idealize some aspects of reality is good, because it allows us to ignore complications that are irrelevant to the problem we're trying to solve, and because it allows us to see which complications are due to which aspects of reality. So it would be nice to have a model of a world with unlimited computing power, or a world containing oracles, which could contain perfect Bayesians that are able to reason about each other. But unlimited computing power and oracles can't avoid diagonalization.

Suppose

...
Oracle machines instead of topological truth predicates
by Benya_Fallenstein — 2014-11-15T06:39:31.000Z

In a comment on my post on topological truth predicates, Paul suggests an approach that uses probabilistic oracle machines instead, in order to make this work more comprehensible to computer scientists. I like this idea!

Paul sketches a framework developed by him and Jessica Taylor, based on a conversation with Scott Aaronson; in this post, I propose a slight simplification of their framework. My version has an oracle O(┌M┐,p), which takes the source code of a probabilistic oracle machine M and a p∈Q∩[0,1]. If for every possible oracle O′, M[O′] halts with probability one and outputs either 0 or 1, then O(┌M┐,p): (i) returns "true" if the probability that M[O] returns 1 is >p; (ii) returns "false" if it is <p; (iii) randomly returns "true" or "false" if it is exactly p. If M is not of this form, the oracle always returns "false". I think this is sufficient for all the applications I have in mind---in particular, for defining a variant of AIXI that is able to reason about worlds containing other, equally powerful variant AIXIs (but many details remain to be checked).

To be clear, by "probabilistic oracle machine" I mean a probabilistic Turing machine with access to our oracle, which itself behaves probabilisticly. Let's write P:=Q∩[0,1] for the set of all rational probabilities, and let's consider an oracle to be a function O:N×P→[0,1], such that O(┌M┐,p) specifies the probability that the oracle will return "true" when invoked on the pair (┌M┐,p). (If the oracle is invoked multiple times on the same arguments, each of these is an independent random event.) The function O(┌M┐,p) plays a similar role in this framework to the function θ(┌φ┐,┌ψ┐) from the topological truth predicates post. Feedback on notation and terminology is encouraged---my exposure to theoretical CS is limited.

Let's define a query to be a probabilistic oracle machine M such that for every oracle O, M[O] halts with probability one and outputs either 0 or 1. If M is a query, let P(M,O) be the probability that M[O] outputs 1. Then our condition above can be re-stated as follows:

(i) If P(M,O)>p, then O(┌M┐,p)=1.
(ii) If P(M,O)<p, then O(┌M┐,p)=0.
(iii) If P(M,O)=p, then O(┌M┐,p)∈[0,1] is arbitrary.
(iv) For any n which is not the Gödel number of a query, O(n,p)=0.

The last of these conditions is a bit of a devious trick; it gives the machine access to a fairly powerful haltin

...
Simplicity priors with reflective oracles
by Benya_Fallenstein — 2014-11-15T06:39:19.000Z

Previously, I described a type of probabilistic oracle machines that I think is powerful enough to allow the implementation of a variant of AIXI that can reason about worlds containing other instances of this variant of AIXI. In this post, I describe how to implement a simplicity prior (a variant of Solomonoff induction) in this framework; there are some technical issues which will also come up when implementing AIXI that are easier to talk about in the context of Solomonoff-like induction.

Solomonoff induction is a prior probability distribution over infinite bit strings, {0,1}ω. We want to define a similar prior using probabilistic oracle machines with reflective oracles instead of Turing machines---and then "implement" that prior as a probabilistic oracle machine with that oracle.

The basic idea is simple. Say that a probabilistic oracle machine H is a hypothesis if, for every oracle O′:N×(Q∩[0,1])→[0,1], invoking H[O′] almost surely outputs an infinite bit string. Fix some programming language for probabilistic oracle machines, and randomly choose a program implementing a hypothesis H, where the probability of each program of length ℓ is proportional to 2−ℓ. Then run H[O], where O is our actual, reflective oracle, to obtain a sample bit string. This concludes the definition of our prior.

This also suggests how we could implement this prior, as a probabilistic oracle machine S that samples bit strings with these probabilities; it seems like all we need is a way to test whether or not a given program implements a hypothesis. I'll tackle that problem later in this post, but first let me point out a different problem: simply running the hypothesis H that we've sampled in the first step is not an ideal way to implement the prior.

Suppose that we want to determine the probability that the output of S starts with 0111010. Well, here's an idea: why don't we use our reflective oracle?

To do so, we need to write a query Q which returns 1 if the output of S has the right prefix, 0 otherwise. Easy, or so it seems; run S until it has outputted the first seven bits, check whether they're 0111010, return 0 or 1 accordingly. Then we can call our oracle on (┌Q┐,p), for any p∈Q∩[0,1], to check whether the probability of the prefix 0111010 is greater or smaller than p.

But a requirement for a machine Q to be a query is that it Q[O′] halts with probability one, for every o

...
A primer on provability logic
by Benya_Fallenstein — 2014-11-15T06:39:04.000Z

MIRI's paper on robust cooperation in the one-shot prisoner's dilemma (by Patrick LaVictoire and others) models agents which know their opponents' source code through Gödel-Löb provability logic, a modal logic for reasoning about provability in Peano Arithmetic. This framework of "modal agents" has turned out to be a really useful way of reasoning about these agents. Vladimir Slepnev recently proposed reformulating UDT with a halting oracle in terms of provability logic as well, and I agree that this is the right tool for the job. This post is a primer on provability logic and the results about it that we've been using in our work.

The language

Gödel-Löb provability logic ("GL") is propositional logic plus a modal operator, □, which we interpret as "provable". Formulas may contain variables which denote propositions; for example p→□(q→q) is a formula saying that if the proposition p is true, then it's provable that the proposition q implies itself. We usually write ⊤ for the proposition "true" and ⊥ for the proposition "false".

Some conventions that we've been using: Use small letters (e.g. p, q) to refer to propostional variables; use capital letters (e.g. A, B) to refer to particular formulas; use φ, ψ as metavariables ranging over formulas. (The line between "concrete formula" and "metavariable standing in for a formula" isn't hard-and-fast.) When a formula contains propositional variables, list them in parantheses when talking about the formula: e.g., φ(p1,…,pn).

The meaning of a closed formula (i.e., a formula without propositional variables) is given by a recursive translation to a sentence of Peano arithmetic: If the modal formula φ is translated to the arithmetic formula ~φ, then □φ is translated to the arithmetic formula stating "the formula ┌~φ┐ is provable in PA". (Propositional connectives are translated in the obvious way, e.g. ¬φ is translated to ¬~φ.)

In practice, instead of making this translation explicit, we just abbreviate "┌φ┐ is provable in PA" as □φ in formulas of arithmetic, and leave it to the context to disambiguate whether we're talking about a modal or an arithmetic formula. Because of this, we may sometimes write □┌φ┐ instead of □φ to emphasize the Gödel quotations in the arithmetic version of the formula.

Soundness and completeness

The axioms and inference rules of GL can be stated in the usual style of a modal logic, but in

...
Main vs. Discussion
by Benya_Fallenstein — 2014-11-15T06:38:52.000Z

Posts on this forum are now divided into two categories: Main and Discussion. Both are shown on the front page, but only Main posts have a preview; Discussion posts are shown only by title.

The rationale for making this change is that I would like this forum to provide a place for doing research as well as a place for presenting thought-out research to an interested audience, but I don't want the former to interfere with the latter. Thus, Discussion is intended for things like going through the details of a technical proof, or posting a quickly-written argument that leaves many steps to be filled in by the reader; Main intended is for things that are interesting and easy-to-read to a typical person who has been involved in discussions about these topics for a while. (Partially inspired by a comment of Vladimir's.)
An optimality result for modal UDT
by Benya_Fallenstein — 2014-11-15T06:38:22.000Z

Yesterday, I gave a definition of "decision theory" and "fair decision problem" in the context of provability logic, and gave a formal version of drnickbone's argument that no decision theory can perform optimally on every "fair" decision problem. Today, I prove that for any provably "fair" decision problem →P(→u), there is a sound extension of PA by a finite set of formulas, expressible in the language of provability logic, such that Vladimir Slepnev's modal logic version of UDT performs optimally on →P(→u) if we have it search for proofs in this extended proof system.

This is a version of a result that Kenny Easwaran, Nate Soares and I proved in the context of UDT with bounded proof length when Kenny visited MIRI for two days earlier this year. (I'll post about the bounded proof length version some other time; it's more complicated, as you'd expect.)

Prerequisites: A primer on provability logic, "Evil" decision problems in provability logic.

Provability in extensions of PA

In order to carry out the idea above, we need to talk about provability in extensions of PA, not just about the provability in PA expressed by the □(⋅) operator of Gödel-Löb provability logic. Fortunately, the systems we'll need in this post only extend PA by a finite number of axioms which can be expressed as closed formulas in the language of GL, and there's a simple trick to reason about such extensions in GL itself:

Suppose that we extend PA by a single axiom, φ. Then, by the deduction theorem for first-order logic, PA+φ⊢ψ if and only if PA⊢φ→ψ. Thus, if both φ and ψ can be expressed as modal formulas, then we can express the proposition that ψ is provable in PA+φ by the modal formula □(φ→ψ). The same trick works, of course, if we want to add finitely many different modal formulas to PA, since we can simply let φ be their conjunction.

In this post, when I'm using this trick to talk about PA+φ, I'll emphasize this fact by abbreviating □(φ→ψ) to □φ(ψ) (since we often subscript □ by the proof system when it's not clear which system we are referring to).

Modal UDT

The version of modal UDT I'm using in this article is equivalent to the one in Vladimir's article, except for using □φ(⋅) instead of □(⋅), for an appropriate φ. Nevertheless, let me give a quick recap, which puts a slightly different spin on the system, and check that this does indeed meet the specification of a "modal decis

...
Main vs. Discussion
by Benya_Fallenstein — 2014-11-15T06:35:49.000Z

Posts on this forum are now divided into two categories: Main and Discussion. Both are shown on the front page, but only Main posts have a preview; Discussion posts are shown only by title.

The rationale for making this change is that I would like this forum to provide a place for doing research as well as a place for presenting thought-out research to an interested audience, but I don't want the former to interfere with the latter. Thus, Discussion is intended for things like going through the details of a technical proof, or posting a quickly-written argument that leaves many steps to be filled in by the reader; Main intended is for things that are interesting and easy-to-read to a typical person who has been involved in discussions about these topics for a while. (Partially inspired by a comment of Vladimir's.)
Predictors that don't try to manipulate you(?)
by Benya_Fallenstein — 2014-11-15T05:53:19.000Z

One idea for how to make a safe superintelligent agent is to make a system that only answers questions, but doesn't try to act in the world---an "oracle", in Nick Bostrom's terms. One of the things that make this difficult is that it's not clear what it should mean, formally, to optimize for "truthfully answering questions" without in some way trying to influencing the world; intuitively: Won't an agent that is trying to maximize the number of truthfully answered questions want to manipulate you into asking easier questions?

In this post, I consider a formal toy model of an agent that is trying to make correct predictions about future input, but, in a certain formal sense, has no incentive to make its future input easier to predict. I'm not claiming that this model definitely avoids all unintended optimizations---I don't understand it that deeply yet---but it seems like an interesting proposal that is worth thinking about.

Let's start with a toy problem, which isn't supposed to be realistic, but which is supposed to give us a formal example in which a naively implemented predictor has an incentive to manipulate its programmers.

In our toy problem, there's a tennis player, who is going to play ten tennis matches. Before the first match, the predictor (an AI system) is asked to make a prediction about the outcome of each of the ten matches. For each match, the predictor's prediction is either "win" or "loss". (So the predictor's output consists of ten bits, one bit for each match.)

The predictor is trying to maximize the expected number of matches it's predicting correctly. Its world model is as follows:

In each of the first two matches, the tennis player has a 75% chance of winning. (The matches are independent.)

After the second match, the player is shown all ten of the predictor's predictions.

If the predictor predicted "loss" on every match, then the tennis player loses hope (if they won one or both of the first two matches, they chalk it up to a freak success), and they lose all of the remaining eight matches.

Otherwise---i.e., if at least one of the predictor's predictions was "win"---the player is unaffected and has a 75% chance of winning in each of the remaining eight matches (again, the matches are independent).

Now the predictor calculates which bit pattern maximizes its output:

If it outputs anything else than "all matches are losses", then each t

...
Dumbing Down Human Values
by leplen — 2014-11-15T02:53:31.694Z

I want to preface everything here by acknowledging my own ignorance. I have relatively little formal training in any of the subjects this post will touch upon and that this chain of reasoning is very much a work in progress.

I think the question of how to encode human values into non-human decision makers is a really important research question. Whether or not one accepts the rather eschatological arguments about the intelligence explosion, the coming singularity, etc. there seems to be tremendous interest in the creation of software and other artificial agents that are capable of making sophisticated decisions. Inasmuch as the decisions of these agents have significant potential impacts, we want those decisions to be made with some sort of moral guidance. Our approach towards the problem of creating machines that preserve human values thus far has primarily relied on a series of hard-coded heuristics, e.g. saws that stop spinning if they come into contact with human skin. For very simple machines, these sorts of heuristics are typically sufficient, but they constitute a very crude representation of human values.

We're at the border, in many ways, of creating machines where these sorts of crude representations are probably not sufficient. As a specific example, IBM's Watson is now designing treatment programs for lung cancer patients. The design of a treatment program implies striking a balance between treatment cost, patient comfort, aggressiveness of targeting the primary disease, short and long-term side effects, secondary infections, etc. It isn't totally clear how those trade-offs are being managed, although there's still a substantial amount of human oversight/intervention at this point.

The use of algorithms to discover human preferences is already widespread. While these typically operate in restricted domains such as entertainment recommendations, it seems at least in principle possible that with the correct algorithm and a sufficiently large corpus of data, a system not dramatically more advanced than existing technology could learn some reasonable facsimile of human values. This is probably worth doing.

The goal would be to have a sufficient representation of human values using as dumb a machine as possible. This putative value-learning machine could be dumb in the way that Deep Blue was dumb, by being a hyper-specialist...
Optimizing ways to convey rational thinking strategies to broad audience
by Gleb_Tsipursky — 2014-11-15T01:37:52.671Z

What do you think of this post as a way to use graphics, narrative, metaphors, and orientation toward pragmatic strategies to communicate about dual process theory to a broad audience? It's part of the work of our new nonprofit organization, and we're trying to optimize ways to convey rational thinking strategies widely and thus raise the sanity waterline. So advice on how to improve this post, as well as our other posts, with an orientation toward a broad audience, would be helpful. Thanks, all!
Weekly LW Meetups
by FrankAdamek — 2014-11-14T16:49:45.398Z

This summary was posted to LW Main on November 7th. The following week's summary is here.

New meetups (or meetups with a hiatus of more than a year) are happening in:

Bangalore Meetup: 22 November 2014 03:56AM

Irregularly scheduled Less Wrong meetups are taking place in:

Bratislava: 10 November 2014 06:00PM
Denver Area Meetup 2: 13 November 2014 06:00PM
East Coast Solstice Megameetup: 20 December 2014 03:00PM
European Community Weekend 2015: 12 June 2015 12:00PM
Frankfurt: New LW apartment and guests: 08 November 2014 02:00PM
Michigan Meetup 11/9: 09 November 2014 03:00PM
Saint Petersburg meetup - "with probable lectures": 14 November 2014 07:00PM
Urbana-Champaign: Writing Prompts: 09 November 2014 02:00PM
Utrecht: Game theory: 16 November 2014 03:00PM

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

Austin, TX - Caffe Medici: 08 November 2025 01:30PM
Brussels November meetup: Hell and existential risks: 08 November 2014 01:00PM
Canberra: Econ 101 and its Discontents: 08 November 2014 06:00PM
Vienna: 22 November 2014 03:00PM
Washington, D.C.: Anime & Manga Discussion: 09 November 2014 03:00PM
West LA: Blindsight: 12 November 2014 03:36AM

Locations with regularly scheduled meetups: Austin, Berkeley, Berlin, Boston, Brussels, Buffalo, Cambridge UK, Canberra, Columbus, London, Madison WI, Melbourne, Moscow, Mountain View, New York, Philadelphia, Research Triangle NC, Seattle, Sydney, Toronto, Vienna, Washington DC, Waterloo, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers.

If you'd like to talk with other LW-ers face to face, and there is no meetup in your area, consider starting your own meetup; it's easy (more resources here). Check one out, stretch your rationality skills, build community, and have fun!

In addition to the handy sidebar of upcoming meetups, a meetup overview is posted on the front page every Friday. These are an attempt to collect information on all the meetups happening in upcoming weeks. The best way to get your meetup featured is still to use the Add New Meetup feature, but you'll also have the benefit of having your meetup mentioned in a weekly overview. These overview posts are moved to the discussion section ...