On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects. The intimate connections of this setting to remote sourcecoding under logarithmic loss distortion measure, information combining, common reconstruction, the WynerAhlswedeKorner problem, the efficiency of investment information, as well as, generalization, variational inference, representation learning, autoencoders, and others are highlighted. We discuss its extension to the distributed information bottleneck problem with emphasis on the Gaussian model and highlight the basic connections to the uplink Cloud Radio Access Networks (CRAN) with oblivious processing. For this model, the optimal tradeoffs between relevance (i.e., information) and complexity (i.e., rates) in the discrete and vector Gaussian frameworks is determined. In the concluding outlook, some interesting problems are mentioned such as the characterization of the optimal inputs (features) distributions under power limitations maximizing the relevance for the Gaussian information bottleneck, under complexity constraints.

On the Difference between the Information Bottleneck and the Deep Information Bottleneck Combining the information bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proven successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the deep variational information bottleneck and the assumptions needed for its derivation. The two assumed properties of the data, X and Y, and their latent representation T, take the form of two Markov chains     T  X  Y     and     X  T  Y    . Requiring both to hold during the optimisation process can be limiting for the set of potential joint distributions     P ( X , Y , T )    . We, therefore, show how to circumvent this limitation by optimising a lower bound for the mutual information between T and Y:     I ( T ; Y )    , for which only the latter Markov chain has to be satisfied. The mutual information     I ( T ; Y )     can be split into two nonnegative parts. The first part is the lower bound for     I ( T ; Y )    , which is optimised in deep variational information bottleneck (DVIB) and cognate models in practice. The second part consists of two terms that measure how much the former requirement     T  X  Y     is violated. Finally, we propose interpreting the family of information bottleneck models as directed graphical models, and show that in this framework, the original and deep information bottlenecks are special cases of a fundamental IB model.