  In distributed learning settings such as federated learning, the training
algorithm can be potentially biased towards different clients. Mohri et al.
(2019) proposed a domain-agnostic learning algorithm, where the model is
optimized for any target distribution formed by a mixture of the client
distributions in order to overcome this bias. They further proposed an
algorithm for the cross-silo federated learning setting, where the number of
clients is small. We consider this problem in the cross-device setting, where
the number of clients is much larger. We propose a communication-efficient
distributed algorithm called Agnostic Federated Averaging (or AgnosticFedAvg)
to minimize the domain-agnostic objective proposed in Mohri et al. (2019),
which is amenable to other private mechanisms such as secure aggregation. We
highlight two types of naturally occurring domains in federated learning and
argue that AgnosticFedAvg performs well on both. To demonstrate the practical
effectiveness of AgnosticFedAvg, we report positive results for large-scale
language modeling tasks in both simulation and live experiments, where the
latter involves training language models for Spanish virtual keyboard for
millions of user devices.
