  With the rapid development of storage and computing power on mobile devices,
it becomes critical and popular to deploy models on devices to save onerous
communication latencies and to capture real-time features. While quite a lot of
works have explored to facilitate on-device learning and inference, most of
them focus on dealing with response delay or privacy protection. Little has
been done to model the collaboration between the device and the cloud modeling
and benefit both sides jointly. To bridge this gap, we are among the first
attempts to study the Device-Cloud Collaborative Learning (DCCL) framework.
Specifically, we propose a novel MetaPatch learning approach on the device side
to efficiently achieve "thousands of people with thousands of models" given a
centralized cloud model. Then, with billions of updated personalized device
models, we propose a "model-over-models" distillation algorithm, namely
MoMoDistill, to update the centralized cloud model. Our extensive experiments
over a range of datasets with different settings demonstrate the effectiveness
of such collaboration on both cloud and device sides, especially its
superiority in modeling long-tailed users.
