  We present counterfactual planning as a design approach for creating a range
of safety mechanisms that can be applied in hypothetical future AI systems
which have Artificial General Intelligence.
  The key step in counterfactual planning is to use an AGI machine learning
system to construct a counterfactual world model, designed to be different from
the real world the system is in. A counterfactual planning agent determines the
action that best maximizes expected utility in this counterfactual planning
world, and then performs the same action in the real world.
  We use counterfactual planning to construct an AGI agent emergency stop
button, and a safety interlock that will automatically stop the agent before it
undergoes an intelligence explosion. We also construct an agent with an input
terminal that can be used by humans to iteratively improve the agent's reward
function, where the incentive for the agent to manipulate this improvement
process is suppressed. As an example of counterfactual planning in a non-agent
AGI system, we construct a counterfactual oracle.
  As a design approach, counterfactual planning is built around the use of a
graphical notation for defining mathematical counterfactuals. This two-diagram
notation also provides a compact and readable language for reasoning about the
complex types of self-referencing and indirect representation which are
typically present inside machine learning agents.
