Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change (2020)

Abstract: The choice of hyper-parameters affects the performance of neural models. While much previous research (Sutskever et al., 2013; Duchi et al., 2011; Kingma and Ba, 2015) focuses on accelerating convergence and reducing the effects of the learning rate, comparatively few papers concentrate on the effect of batch size. In this paper, we analyze how increasing batch size affects gradient direction, and propose to evaluate the stability of gradients with their angle change. Based on our observations, the angle change of gradient direction first tends to stabilize (i.e. gradually decrease) while accumulating mini-batches, and then starts to fluctuate. We propose to automatically and dynamically determine batch sizes by accumulating gradients of mini-batches and performing an optimization step at just the time when the direction of gradients starts to fluctuate. To improve the efficiency of our approach for large models, we propose a sampling approach to select gradients of parameters sensitive to the batch size. Our approach dynamically determines proper and efficient batch sizes during training. In our experiments on the WMT 14 English to German and English to French tasks, our approach improves the Transformer with a fixed 25k batch size by +0.73 and +0.82 BLEU respectively.

## introduction the performance of neural models is likely to be affected by the choice of hyper-parameters. while much previous research @xcite @xcite focuses on accelerating convergence and reducing the effects of the learning rate, comparatively few papers concentrate on the effect of batch size. however, batch size is also an important hyperparameter, and some batch sizes empirically lead to better performance than the others. specifically, it has been shown that the performance of the transformer model @xcite for neural machine translation (nmt) @xcite @xcite relies heavily on the batch size @xcite @xcite . the influence of batch size on performance raises the question, how to dynamically find proper and efficient batch sizes during training? in this paper, we investigate the relationship between the batch size and gradients, and propose a dynamic batch size approach by monitoring gradient direction changes. our contributions are as follows: ## conclusion in this paper, we analyze the effects of accumulated batches on the gradient direction, and propose to achieve efficient automated batch sizes by monitoring change in gradient accumulation and performing an optimization step when the accumulated gradient direction is almost stable. to improve the efficiency of our approach with large models, we propose a sampling approach to select gradients of parameters sensitive to the batch size. our approach improves the transformer with a fixed 25k batch size by +0.73 and +0.82 bleu on the wmt 14 english to german and english to french tasks respectively while preserving efficiency.