trentmkelly/LessWrong-43k huggingface dataset:
In defense of Oracle ("Tool") AI research
by Steven Byrnes — 2019-08-07T19:14:10.435Z

(Update 2022: Enjoy the post, but note that it’s old, has some errors, and is certainly not reflective of my current thinking. –Steve)

Low confidence; offering this up for discussion

An Oracle AI is an AI that only answers questions, and doesn't take any other actions. The opposite of an Oracle AI is an Agent AI, which might also send emails, control actuators, etc.

I'm especially excited about the possibility of non-self-improving oracle AIs, dubbed Tool AI in a 2012 article by Holden Karnofsky.

I've seen two arguments against this "Tool AI":

First, as in Eliezer's 2012 response to Holden, we don't know how to safely make and operate an oracle AGI (just like every other type of AGI). Fair enough! I never said this is an easy solution to all our problems! (But see my separate post for why I'm thinking about this.)

Second, as in Gwern's 2016 essay, there's a coordination problem. Even if we could build a safe oracle AGI, the argument goes, there will still be an economic incentive to build an agent AGI, because you can do more and better and faster by empowering the AGI to take actions. Thus, agreeing to never ever build agent AGIs is a very hard coordination problem for society. I don't find the coordination argument compelling—in fact, I think it's backwards—and I wrote this post to explain why.

Five reasons I don't believe the coordination / competitiveness argument against oracles

1. If the oracle isn't smart or powerful enough for our needs, we can solve that by bootstrapping. Even if the oracle is not inherently self-modifying, we can ask it for advice and do human-in-the-loop modifications to make more powerful successor oracles. By the same token, we can ask an oracle AGI for advice about how to design a safe agent AGI.

2. Avoiding coordination problems is a pipe dream; we need to solve the coordination problem at some point, and that point might as well be at the oracle stage. As far as I can tell, we will never get to a stage where we know how to build safe AGIs and where there is no possibility of making more-powerful-and-less-safe AGIs. If we have a goal in the world that we really really want to happen, a low-impact agent is going to be less effective than a not-impact-restrained agent; an act-based agent is going to be less effective than a goal-seeking agent;[1] and so on and so forth. It seems likely that, no matter how powerful a safe AGI we can...
Help forecast study replication in this social science prediction market
by rosiecam — 2019-08-07T18:18:03.102Z

This is a linkpost for: https://www.replicationmarkets.com/

From their sign-up page:

Help the Replication Markets team spot reliable science, and win $$!

From mid-2019 to mid-2020, we will run prediction markets on 3,000 recently-published social and behavioral science claims. We want you to forecast the outcomes of possible replication trials. On average, only about 50% of published social science results replicate, but it’s not a coin toss: four previous markets have been about 75% accurate. We think we can beat 80%, while forecasting 10-100x as many claims.

Join us to improve social and behavioral science, try new kinds of markets and surveys, and earn $$ prizes. (Over $100,000 in total prizes, distributed via Google Pay, among a target pool of 500 forecasters based on accuracy and contributions.)

Please look around, see the Replication Markets Home Page for more details, or click “Sign Up” to join.
Self-Supervised Learning and AGI Safety
by Steven Byrnes — 2019-08-07T14:21:37.739Z

(Update later on: I'm no longer personally excited about this direction, for reasons mentioned in section 7.2 here.)

Abstract: We should seriously consider the possibility that we'll build AGIs by self-supervised learning. If so, the AGI safety issues seem to be in many respects different (and I think more promising) than in the usual reinforcement learning paradigm. In particular, I'll propose that if we follow certain constraints, these systems can be turned into safe, unambitious AGI oracles. I'll end with lots of open questions.

Epistemic status: Treat as brainstorming. This post supersedes my previous post on this general topic, The Self-Unaware AI Oracle for reasons mentioned below.

What is self-supervised learning, and why might it lead to AGI?

Self-supervised learning consists of taking data, masking off part of it, and training an ML system to use the unmasked data to predict the masked data. To make the predictions better and better, the system needs to develop an increasingly deep and comprehensive semantic understanding of the world. For example, say there's a movie with a rock falling towards the ground. If you want to correctly predict what image will be in the frame a few seconds ahead, you need to predict that the rock will stop when it gets to the ground, not continue falling. Or if there's a movie with a person saying "I'm going to sit down now", you need to predict that they might well sit in a chair, and probably won't start dancing.

Thus, predicting something has close relationship with understanding it. Imagine that you're in a lecture class on a topic where you're already an expert. As the professor is talking, you feel you can often finish their sentences. By contrast, when you're lost in a sea of jargon you don't understand, you can only finish their sentences with a much wider probability distribution ("they're probably going to say some jargon word now").

The term "self-supervised learning" (replacing the previous and more general term "unsupervised learning") seems to come from Yann LeCun, chief AI scientist at Facebook and co-inventor of CNNs. As he wrote here:

I now call it "self-supervised learning", because "unsupervised" is both a loaded and confusing term.

In self-supervised learning, the system learns to predict part of its input from other parts of it input. In other words a portion of the input is used as a supervisory signal

...
Emotions are not beliefs
by Chris_Leong — 2019-08-07T06:27:49.812Z

This lesson took me a long time to learn. Consider the following questions:

Should I be confident? Do you regret that decision? Are you happy about what happened?

In each case, there are two things that are being asked:

Do I have a good chance of success? AND Should I feel confident?
Would you change the decision if you had the opportunity? AND Are you feeling negative emotions about having made that decision?
Was the outcome in line with your preferences? AND Is the outcome creating positive emotional affect for you?

Because these are two seperate questions, they can have two separate answers. It's not common, but you can believe that you are unlikely to succeed, whilst also having the internal emotional experience associated with confidence. You could wish that you had made a different decision without necessarily feeling negative emotions because of it. You can have an outcome that isn't in line with your preferences, but experience positive emotional affect by looking on the bright side.

Truth

Properties like true or false only strictly apply to beliefs. Whether or not you are likely to succeed is either true or false, but this doesn't apply to the emotive affect of confidence itself. Someone can affectively feel very "confident" without being factually wrong.

Calibration

Nonetheless, explicit knowledge isn't the only knowledge contained in our brains. We also possess implicit knowledge and this can be calibrated or uncalibrated. Since these are heuristics, they can't literally be true or false or it would no longer be a heuristic. But we can create different measures of how accurate these heuristics are.

Emotions are linked to our heuristics, so we don't always want to shift ourselves towards feeling positive affect as this might bias us. However, emotions are seperate from implicit knowledge. For example, someone can have implicit knowledge that something is a bad idea, without experiencing negative affect like a sense of dread. Alternatively, someone can have a sense of dread, yet also have a strong intuition that that it is the right decision. Since they are separate, they'll be times when we can experience positive affect instead of the "appropriate" emotion without having a significantly detrimental impact on our calibration.

Unfortunately, it is very hard to predict to what extent emotions affect our implicit understanding. This means it is very hard to figur...
Understanding Recent Impact Measures
by Matthew Barnett — 2019-08-07T04:57:04.352Z

In the first five years after Stuart Armstrong posted his first research suggestions for impact measures, very little published work expanded on the idea. The last post in this sequence was intended to somewhat comprehensively review this literature, but it surveyed only four papers total, including the original article.

In the last two years, research has now picked up pace by a significant margin. The two papers which are most significant are Penalizing side effects using stepwise relative reachability by Victoria Krakovna et al. and Conservative Agency by Alexander Turner et al. In that time a few blog posts have come out explaining the approaches in more detail, and public debate over the utility of impact measures has become much more visible.

Here I will briefly explain the two most prominent measures, relative reachability and attainable utility. We will see that they diverge conceptually from earlier research. By being different, they also end up satisfying some desirable properties. I will then consider some recent notable critiques of impact measures more generally. A personal analysis of these critiques will wait one more day. This post will only cover the surface.

Preliminaries

Before I can explain either of the two measures, I must first introduce the language which allows me to precisely define each approach. Both impact measures have quite simple natural language descriptions, but it is easy to feel as though one is not getting the full story if it is explained using English alone. The specific way that the two methods are represented takes place within a Markov decision process (MDP).

Intuitively, an MDP is just a way of representing actions that an agent can take in a stochastic environment, which is made up of a set of states. Formally, an MDP is defined by a tuple (S,A,r,p,γ). S is the set of states in the environment. A is the set of actions that the agent can take. r is a function which maps state-action pairs to a real number reward. p is a function which returns the probability of transitioning into one state given the previous state and an action, p(st+1|st,at). γ is the discount factor for the rewards, γ∈[0,1].

Relative reachability

In Victoria Krakovna's blog post introducing relative reachability, she explains that relative reachability was a synthesis of two related ideas: preserving reversibility of the environment, and penalizing ...
[Site Update] Behind the scenes data-layer and caching improvements
by habryka — 2019-08-07T00:49:29.721Z

We just pushed a major behind the scenes update we've been working on for a while. This was primarily about reducing technical debt and upgrading a bunch of our outdated dependencies, so you shouldn't see much happening on your side besides some small performance improvements and a general reduction in weird data-loading jankiness.

This was however a pretty big refactor for our codebase, making this a very high risk update. We've spent a lot of time testing things, but please let us know if you notice anything broken or weird in the next few days.
Project Proposal: Considerations for trading off capabilities and safety impacts of AI research
by David Scott Krueger (formerly: capybaralet) — 2019-08-06T22:22:20.928Z

There seems to be some amount of consensus that people working on AI safety (at least within fairly mainstream ML / AI paradigms) shouldn't worry much about the effects of their projects on AI capabilities. New researchers might even try to push capabilities research forward, to build career capital. The best argument for this, IMO, is that someone else is probably going to do it if you don't, likely within the next 6 months (given the current pace of research).

I mostly agree with this view, but I do still think a bit about effects of my research on capabilities, and think others should as well. Being concerned about advancing capabilities has, in the past, moved me away from pursuing ambitious capabilities projects which might have been very good for my career if they paid off, but I always saw someone else do the thing I was considering soon afterwards anyways...

But as far as I know, nobody has tried to evaluate this question thoroughly and systematically. This is concerning, because it seems like current attitudes could plausibly be a result of motivated reasoning (i.e. "I want to keep doing my research, and probably would do so even if I saw a compelling case against it") and groupthink ("nobody else is worrying about this"). I'm not sure it's really tractable, but I think it could be worth ~1-4 people spending a bit of time (possibly up to ~6-24 months, if it ends up looking tractable after some initial thought/investigation) on trying to do a fairly comprehensive treatment of this question.

The main deliverables could be practical guidelines for AI safety researchers, e.g.:

Figuring out when it makes sense to be concerned about advancing AI capabilities via ones' research.
How to decide how significant those concerns are, and whether they should preclude working on that line of research or research project, or change the publication model for them.

The project could intersect with current "dual-use" considerations (e.g. RE GPT-2).

(also worth mentioning): I know MIRI now has secret research, and I think they have a reasonable case for that, since they aren't in the mainstream paradigms. I do think it would be good for them to have a "hit publication" within the ML community, and might be worth pushing some out-of-the-box ideas which might advance capabilities. The reason is that...
Subagents, neural Turing machines, thought selection, and blindspots
by Kaj_Sotala — 2019-08-06T21:15:24.400Z

In my summary of Consciousness and the Brain (Dehaene, 2014), I briefly mentioned that one of the functions of consciousness is to carry out artificial serial operations; or in other words, implement a production system (equivalent to a Turing machine) in the brain.

While I did not go into very much detail about this model in the post, I’ve used it in later articles. For instance, in Building up to an Internal Family Systems model, I used a toy model where different subagents cast votes to modify the contents of consciousness. One may conceptualize this as equivalent to the production system model, where different subagents implement different production rules which compete to modify the contents of consciousness.

In this post, I will flesh out the model a bit more, as well as applying it to a few other examples, such as emotion suppression, internal conflict, and blind spots.

Evidence accumulation

Dehaene has outlined his model in a pair of papers (Zylberberg, Dehaene, Roelfsema, & Sigman, 2011; Dehaene & Sigman, 2012), though he is not the first one to propose this kind of a model. Daniel Dennett’s Consciousness Explained (1991) also discusses consciousness as implementing a virtual Turing machine; both cite as examples earlier computational models of the mind, such as Soar and ACT, which work on the same principles.

An important building block in Dehane’s model is based on what we know about evidence accumulation and decision-making in the brain, so let’s start by taking a look at that.

Sequential sampling models (SSMs) are a family of models from mathematical psychology that have been developed since the 1960s (Forstmann, Ratcliff, & Wagenmakers, 2016). A particularly common SSM is the diffusion decision model (DDM) of decision-making, in which a decision-maker is assumed to noisily accumulate evidence towards a particular choice. Once the evidence in favor of a particular choice meets a decision threshold, that choice is taken.

For example, someone might be shown dots on a screen, some of which are moving in a certain direction. The task is to tell which direction the dots are moving in. After the person has seen enough dot movements, they will have sufficient confidence to make their judgment. The difficulty of the task can be precisely varied by changing the proportion of moving dots and their speed, making the move...
Percent reduction of gun-related deaths by color of gun.
by Gunnar_Zarncke — 2019-08-06T20:28:56.134Z

A thought experiment: Assume all guns were pink by law tomorrow.

Would that have an impact on the number of gun-related deaths? What percentage?

Would it have an impact on the number of mass shootings?

This is not an actual policy suggestion, so the feasibility of such a policy is irrelevant. It is more a question of the psychological impact of the color on the behavior of gun users (otherwise the usage of the color pink stays the same). Choose other colors if that's relevant for your answer.

Disclaimer: This is not my idea and you can quickly find out who asked it on twitter, but there are no answers on twitter yet, so better google somewhere else (also to avoid politics the mind-killer). I will reveal the source of the question later in the comments.
New paper: Corrigibility with Utility Preservation
by Koen.Holtman — 2019-08-06T19:04:26.386Z

I am pleased to announce the availability of a long-format paper with new results on AGI safety: Corrigibility with Utility Preservation.
You can get the paper at https://arxiv.org/abs/1908.01695 , and in the related software repository at https://github.com/kholtman/agisim .
Abstract
Corrigibility is a safety property for artificially intelligent agents. A corrigible agent will not resist attempts by authorized parties to alter the goals and constraints that were encoded in the agent when it was first started. This paper shows how to construct a safety layer that adds corrigibility to arbitrarily advanced utility
maximizing agents, including possible future agents with Artificial General Intelligence (AGI). The layer counter-acts the emergent incentive of advanced agents to resist such alteration.
A detailed model for agents which can reason about preserving their utility function is developed, and used to prove that the corrigibility layer works as intended in a large set of non-hostile universes. The corrigible agents have an emergent incentive to protect key elements of their corrigibility layer. However, hostile universes may contain forces strong enough to break safety features. Some open problems related to graceful degradation when an agent is successfully attacked are identified.
The results in this paper were obtained by concurrently developing an AGI agent simulator, an agent model, and proofs. The simulator is available under an open source license. The paper contains simulation results which illustrate the safety related properties of corrigible AGI agents in detail.
This post can be used for comments and questions.
The paper contains several results and observations that do not rely on the heavy use of math, but other key results and discussions are quite mathematical. Feel to post questions and comments even if you have not read all the mathematical parts.
As this is my first post on LessWrong, and my first paper on AGI safety, I feel I should say something to introduce myself. I have a Ph.D. in software design, but my professional life so far has been very diverse and multidisciplinary. Among other things I have been an experimental physicist, a standards developer and negotiator, an Internet privacy advocate, a wireless networking expert, and a systems architect in an industrial research lab. So I bring a wide range of tools and methodologic...
Weak foundation of determinism 
 analysis
by aiiixiii — 2019-08-06T19:03:40.982Z

Determinism is the belief that every action in time is born by the previous one. [1] In deterministic terms you cannot have an event E if you didn't have an event D before, which in turn is the result of a cause C and so on, without any type of alphabetical bound. This philosophical conception is one of the most discussed, directly or indirectly, because it is the generator of very important ontological implications, first of all the existence of free will which, we could claim, is one of the most courted topics by the philosophers of all time. Something that has always struck me about the endless debates on the subject it is a simple logical flaw that people seem to commit when they put forward their arguments, both for and against determinism. This (ir)rational weakness lies in the very concept core of determinism itself and, for simplicity, I will call it the "What if ?" problem. It goes like this:

" If determinism is true, then should criminals be persecuted for their crimes ? "
" If determinism is true, then should science cease to exists ? "
" Why, if determinism is true, do we look carefully before crossing the road ? "
" If determinism is true why should I do anything ? "

I think you got the point. Now, questions like these can be found in countless publications, online blog discussions, talk with friends on a drug-induced Friday night and they generally give way to endless verbosity flows, some of which may also contain angular viewpoints that shed new lights on your beliefs. So far so beautiful, too bad all these intellectual disputes are inconsistent with the very premise of determinism. Think of the first question, the only correct answer is:

"If determinism is true, criminals will continue to commit crimes and we will condemn them for this, we could not change our way of acting because otherwise it would not be determinism."

The second answer is a reflection of this and also the third. In fact, this answer is a blu...