We introduce Dynamic Deep Neural Networks (D 2 NN), a new type of feed-forward deep neural network that allow selective execution. Given an input, only a subset of D 2 NN neurons are executed, and the particular subset is determined by the D 2 NN itself. By pruning unnecessary computation depending on input, D 2 NNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a D 2 NN augments a regular feed-forward deep neural network (directed acyclic graph of differentiable modules) with one or more controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A D 2 NN is trained end to end. Both regular modules and controller modules in a D 2 NN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various D 2 NN architectures on image classification tasks, we demonstrate that D 2 NNs are general and flexible, and can effectively optimize accuracy-efficiency trade-offs.
The closest work to ours is Dynamic Deep Neural Networks REF which allows selective execution by integrating backpropagation with reinforcement learning.
Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
ArXiv
ArXiv
Computer Science Mathematics