Augment your batch: better training with larger batches
Authors: Elad Hoffer, Itay Hubara, Niv Giladi, Daniel Soudry
Keywords: Large Batch Training, Augmentation, Deep Learning
TL;DR: Improve accuracy by large batches composed of multiple instances of each sample at the same batch
Abstract: Recently, there is regained interest in large batch training of neural networks, both of theory and practice. New insights and methods allowed certain models to be trained using large batches with no adverse impact on performance. Most works focused on accelerating wall clock training time by modifying the learning rate schedule, without introducing accuracy degradation. 
We propose to use large batch training to boost accuracy and accelerate convergence by combining it with data augmentation. Our method, "batch augmentation", suggests using multiple instances of each sample at the same large batch. We show empirically that this simple yet effective method improves convergence and final generalization accuracy. We further suggest possible reasons for its success.