State-of-the-art machine learning systems rely on graph-based models, with the distributed training of these models being the norm in AI-powered production pipelines. The performance of these communication-heavy systems depends on the effective overlap of communication and computation. While the overlap challenge has been addressed in systems with simpler model representations, it remains an open problem in graph-based models . In this work, we  develop a system for communication scheduling which realizes near-optimal overlap of communication and computation in graph-based models. Our system is implemented over TensorFlow and requires no changes in the model or developer inputs. Our system improves the throughput by up to 82\% in inference and 20\% in training, while also reducing straggler effect by up to 2.8x. A part of our implementation is already merged with TensorFlow codebase; the rest is publicly available.%DIFDELCMD < \end{abstract} %DIFDELCMD < %%% \\%DIF > in inference and 19.2\\% in training, while also reducing straggler effect by up to 2.3\times. Our code is publicly available.
State-of-the-art deep learning systems rely on iterative distributed training to tackle the increasing complexity of models and input data. The iteration time in these communication-heavy systems depends on the computation time, communication time and the extent of overlap of computation and communication . In this work, we identify a shortcoming in systems with graph representation for computation, such as TensorFlow and PyTorch, that result in high variance in iteration time --- random order of received parameters across workers. We develop a system , TicTac, to improve the iteration time by fixing this issue in distributed deep learning with Parameter Servers while guaranteeing near-optimal overlap of communication and computation . TicTac identifies and enforces an order of network transfers which improves the iteration time using prioritization. Our system is implemented over TensorFlow and requires no changes to the model or developer inputs. TicTac improves the throughput by up to %DIFDELCMD < \end{abstract} %DIFDELCMD < %%% 37.7\\%DIF > in inference and 19.2\\% in training, while also reducing straggler effect by up to 2.3\times. Our code is publicly available.
R
machine
deep
R
graph-based models, with the distributed training of these models being the norm in AI-powered production pipelines. The performance of
iterative distributed training to tackle the increasing complexity of models and input data. The iteration time in
R
effective overlap of communication and computation. While the overlap challenge has been addressed in systems with simpler model representations, it remains an open problem in graph-based models
computation time, communication time and the extent of overlap of computation and communication
A
identify a shortcoming in systems with graph representation for computation, such as TensorFlow and PyTorch, that result in high variance in iteration time --- random order of received parameters across workers. We
R
for communication scheduling which realizes
, TicTac, to improve the iteration time by fixing this issue in distributed deep learning with Parameter Servers while guaranteeing
R
in graph-based models.
. TicTac identifies and enforces an order of network transfers which improves the iteration time using prioritization.
R
in
to
R
Our system
TicTac
D
82\% in inference and 20\% in training, while also reducing straggler effect by up to 2.8x. A part of our implementation is already merged with TensorFlow codebase; the rest is publicly available.
A
37.7