  The performance of federated learning systems is bottlenecked by
communication costs and training variance. The communication overhead problem
is usually addressed by three communication-reduction techniques, namely, model
compression, partial device participation, and periodic aggregation, at the
cost of increased training variance. Different from traditional distributed
learning systems, federated learning suffers from data heterogeneity (since the
devices sample their data from possibly different distributions), which induces
additional variance among devices during training. Various variance-reduced
training algorithms have been introduced to combat the effects of data
heterogeneity, while they usually cost additional communication resources to
deliver necessary control information. Additionally, data privacy remains a
critical issue in FL, and thus there have been attempts at bringing
Differential Privacy to this framework as a mediator between utility and
privacy requirements. This paper investigates the trade-offs between
communication costs and training variance under a resource-constrained
federated system theoretically and experimentally, and how communication
reduction techniques interplay in a differentially private setting. The results
provide important insights into designing practical privacy-aware federated
learning systems.
