Data shuffling is one of the fundamental building blocks for distributed learning algorithms, that increases the statistical gain for each step of the learning process. In each iteration, different shuffled data points are assigned by a central node to a distributed set of workers to perform local computations, which leads to communication bottlenecks. The focus of this paper is on formalizing and understanding the fundamental information-theoretic tradeoff between storage (per worker) and the worst-case communication overhead for the data shuffling problem. We completely characterize the information theoretic tradeoff for K = 2, and K = 3 workers, for any value of storage capacity, and show that increasing the storage across workers can reduce the communication overhead by leveraging coding. We propose a novel and systematic data delivery and storage update strategy for each data shuffle iteration, which preserves the structural properties of the storage across the workers, and aids in minimizing the communication overhead in subsequent data shuffling iterations. Related Work: In the past few years, there has been a flurry of research acitivity in understanding the benefits of coding for caching starting from the work of Maddah-Ali and Niesen [6] who showed that exploiting multi-casting opportunities by coding can reduce the communication for caching. In [7], coding for MapReduce was proposed in order to reduce the communication cost between mappers and reducers, however the underlying focus of [7] is significantly different than the problem considered in this work, where we care about the communication between the master node and the workers. The paper most closely related to this work is [8] , where the idea of coding for data shuffling problem is presented to reduce the communication overhead between the master node and worker nodes. [8] provides a probabilistic scheme of leveraging coding based on a random storage placement. In contrast to [8] , in this paper we provide a deterministic and systematic storage update scheme, which increases the coding opportunities in the delivery phase. The underlying metric used here is the worstcase communication cost over all the possible shuffles, unlike the average cost considered in [8] . Finally, we also present the first information theoretic lower bounds on the communication overhead for the data shuffling problem.
In a related word to coded data shuffling, REF studies the information theoretic limits of data shuffling in distributed learning.
Information Theoretic Limits of Data Shuffling for Distributed Learning
2016 IEEE Global Communications Conference (GLOBECOM)
2016 IEEE Global Communications Conference (GLOBECOM)
Computer Science Mathematics