The Conditional Entropy Bottleneck
Authors: Ian Fischer
Keywords: representation learning, information theory, uncertainty, out-of-distribution detection, adversarial example robustness, generalization, objective function
TL;DR: The Conditional Entropy Bottleneck is an information-theoretic objective function for learning optimal representations.
Abstract: We present a new family of objective functions, which we term the Conditional Entropy Bottleneck (CEB). These objectives are motivated by the Minimum Necessary Information (MNI) criterion. We demonstrate the application of CEB to classification tasks. We show that CEB gives: well-calibrated predictions; strong detection of challenging out-of-distribution examples and powerful whitebox adversarial examples; and substantial robustness to those adversaries. Finally, we report that CEB fails to learn from information-free datasets, providing a possible resolution to the problem of generalization observed in Zhang et al. (2016).