  Today's robotic fleets are increasingly measuring high-volume video and LIDAR
sensory streams, which can be mined for valuable training data, such as rare
scenes of road construction sites, to steadily improve robotic perception
models. However, re-training perception models on growing volumes of rich
sensory data in central compute servers (or the "cloud") places an enormous
time and cost burden on network transfer, cloud storage, human annotation, and
cloud computing resources. Hence, we introduce HarvestNet, an intelligent
sampling algorithm that resides on-board a robot and reduces system bottlenecks
by only storing rare, useful events to steadily improve perception models
re-trained in the cloud. HarvestNet significantly improves the accuracy of
machine-learning models on our novel dataset of road construction sites, field
testing of self-driving cars, and streaming face recognition, while reducing
cloud storage, dataset annotation time, and cloud compute time by between
65.7-81.3%. Further, it is between 1.05-2.58x more accurate than baseline
algorithms and scalably runs on embedded deep learning hardware. We provide a
suite of compute-efficient perception models for the Google Edge Tensor
Processing Unit (TPU), an extended technical report, and a novel video dataset
to the research community at https://sites.google.com/view/harvestnet.
