  Deep learning models in large-scale machine learning systems are often
continuously trained with enormous data from production environments. The sheer
volume of streaming training data poses a significant challenge to real-time
training subsystems and ad-hoc sampling is the standard practice. Our key
insight is that these deployed ML systems continuously perform forward passes
on data instances during inference, but ad-hoc sampling does not take advantage
of this substantial computational effort. Therefore, we propose to record a
constant amount of information per instance from these forward passes. The
extra information measurably improves the selection of which data instances
should participate in forward and backward passes. A novel optimization
framework is proposed to analyze this problem and we provide an efficient
approximation algorithm under the framework of Mini-batch gradient descent as a
practical solution. We also demonstrate the effectiveness of our framework and
algorithm on several large-scale classification and regression tasks, when
compared with competitive baselines widely used in industry.
