  Recently, deep neural networks have become to be used in a variety of
applications. While the accuracy of deep neural networks is increasing, the
confidence score, which indicates the reliability of the prediction results, is
becoming more important. Deep neural networks are seen as highly accurate but
known to be overconfident, making it important to calibrate the confidence
score. Many studies have been conducted on confidence calibration. They
calibrate the confidence score of the model to match its accuracy, but it is
not clear whether these confidence scores can improve the performance of
systems that use confidence scores. This paper focuses on cascade inference
systems, one kind of systems using confidence scores, and discusses the desired
confidence score to improve system performance in terms of inference accuracy
and computational cost. Based on the discussion, we propose a new confidence
calibration method, Learning to Cascade. Learning to Cascade is a simple but
novel method that optimizes the loss term for confidence calibration
simultaneously with the original loss term. Experiments are conducted using two
datasets, CIFAR-100 and ImageNet, in two system settings, and show that naive
application of existing calibration methods to cascade inference systems
sometimes performs worse. However, Learning to Cascade always achieves a better
trade-off between inference accuracy and computational cost. The simplicity of
Learning to Cascade allows it to be easily applied to improve the performance
of existing systems.
