The Simple Essence of Automatic Differentiation Automatic differentiation (AD) in reverse mode (RAD) is a central component of deep learning and other uses of largescale optimization. Commonly used RAD algorithms such as backpropagation, however, are complex and stateful, hindering deep understanding, improvement, and parallel execution. This paper develops a simple, generalized AD algorithm calculated from a simple, natural specification. The general algorithm can be specialized by varying the representation of derivatives. In particular, applying wellknown constructions to a naive representation yields two RAD algorithms that are far simpler than previously known. In contrast to commonly used RAD implementations, the algorithms defined here involve no graphs, tapes, variables, partial derivatives, or mutation. They are inherently parallelfriendly, correct by construction, and usable directly from an existing programming language with no need for new data types or programming style, thanks to use of an ADagnostic compiler plugin.

Constrained MultiObjective Optimization for Automated Machine Learning Automated machine learning has gained a lot of attention recently. Building and selecting the right machine learning models is often a multiobjective optimization problem. General purpose machine learning software that simultaneously supports multiple objectives and constraints is scant, though the potential benefits are great. In this work, we present a framework called Autotune that effectively handles multiple objectives and constraints that arise in machine learning problems. Autotune is built on a suite of derivativefree optimization methods, and utilizes multilevel parallelism in a distributed computing environment for automatically training, scoring, and selecting good models. Incorporation of multiple objectives and constraints in the model exploration and selection process provides the flexibility needed to satisfy tradeoffs necessary in practical machine learning applications. Experimental results from standard multiobjective optimization benchmark problems show that Autotune is very efficient in capturing Pareto fronts. These benchmark results also show how adding constraints can guide the search to more promising regions of the solution space, ultimately producing more desirable Pareto fronts. Results from two realworld case studies demonstrate the effectiveness of the constrained multiobjective optimization capability offered by Autotune.