  The Information Bottleneck (IB) provides an information theoretic principle
for representation learning, by retaining all information relevant for
predicting label while minimizing the redundancy. Though IB principle has been
applied to a wide range of applications, its optimization remains a challenging
problem which heavily relies on the accurate estimation of mutual information.
In this paper, we present a new strategy, Variational Self-Distillation (VSD),
which provides a scalable, flexible and analytic solution to essentially
fitting the mutual information but without explicitly estimating it. Under
rigorously theoretical guarantee, VSD enables the IB to grasp the intrinsic
correlation between representation and label for supervised training.
Furthermore, by extending VSD to multi-view learning, we introduce two other
strategies, Variational Cross-Distillation (VCD) and Variational
Mutual-Learning (VML), which significantly improve the robustness of
representation to view-changes by eliminating view-specific and task-irrelevant
information. To verify our theoretically grounded strategies, we apply our
approaches to cross-modal person Re-ID, and conduct extensive experiments,
where the superior performance against state-of-the-art methods are
demonstrated. Our intriguing findings highlight the need to rethink the way to
estimate mutual
