  Decentralized training of deep learning models is a key element for enabling
data privacy and on-device learning over networks. In realistic learning
scenarios, the presence of heterogeneity across different clients' local
datasets poses an optimization challenge and may severely deteriorate the
generalization performance.
  In this paper, we investigate and identify the limitation of several
decentralized optimization algorithms for different degrees of data
heterogeneity. We propose a novel momentum-based method to mitigate this
decentralized training difficulty. We show in extensive empirical experiments
on various CV/NLP datasets (CIFAR-10, ImageNet, AG News, and SST2) and several
network topologies (Ring and Social Network) that our method is much more
robust to the heterogeneity of clients' data than other existing methods, by a
significant improvement in test performance ($1\% \!-\! 20\%$).
