Representation Learning with Conditional Information Flow Maximization (2024)

Abstract: This paper proposes an information-theoretic representation learning framework, named conditional information flow maximization, to extract noise-invariant sufficient representations for the input data and target task. It promotes the learned representations have good feature uniformity and sufficient predictive ability, which can enhance the generalization of pre-trained language models (PLMs) for the target task. Firstly, an information flow maximization principle is proposed to learn more sufficient representations for the input and target by simultaneously maximizing both input-representation and representation-label mutual information. Unlike the information bottleneck, we handle the input-representation information in an opposite way to avoid the over-compression issue of latent representations. Besides, to mitigate the negative effect of potential redundant features from the input, we design a conditional information minimization principle to eliminate negative redundant features while preserve noise-invariant features. Experiments on 13 language understanding benchmarks demonstrate that our method effectively improves the performance of PLMs for classification and regression. Extensive experiments show that the learned representations are more sufficient, robust and transferable.

## introduction the goal of deep representation learning @xcite is to transform the raw observational data into low-dimensional representations that are essential for various downstream tasks. in recent years, information-theoretic representation learning has been widely studied, aiming to discover useful representations in a principled manner. the infomax principle @xcite has extensive applications in the field of self-supervised representation learning @xcite @xcite . in supervised scenarios, minimizing the standard cross-entropy is actually equivalent to maximizing the mutual information between the representations and the target task @xcite . but infomax tends to preserve potential redundant features that are irrelevant to the given target, leading to biased representations. another noteworthy line of information-theoretic research is built upon the information bottleneck (ib) principle @xcite , which aims to discover compact and informative representations that can reduce redundant features from the inputs @xcite @xcite @xcite . ib seeks to find a maximally compressed representation of the input that preserves as much information as possible about the target, striking a balance between compression and prediction. however, in the information flow @xcite of neural networks, directly reducing the mutual information between the input x and representations z would violate the sufficiency constraint, and may lose the necessary information for the target task y . under the markov chain constraint @xmath0, it's hard to determine beforehand how close we are to optimal compression, and this can easily lead to the over-compression issue of latent representations @xcite . as a result, current ib-based methods would yield insufficient representations for the target task, and hamper prediction ability of neural networks. to ensure sufficiency for the target task and mitigate the negative effect of redundant features, we propose a principled representation learning framework, named conditional information flow maximization (cifm), to extract noise-invariant sufficient representations for the input and target. it promotes the learned representations have good feature uniformity and sufficient predictive ability, which can enhance the generalization of pre-trained language models (plms) for the target task. firstly, we propose an information flow maximization (ifm) principle to learn more sufficient representations for the input and target. it simultaneously maximizes both input-representation and representation-label mutual information. maximizing input-representation information i(x; z) ensures sufficiency of the representations for the input x and preserves information relevant to the target y , and maximizing representation-label information i(y ; z) captures necessary information relevant to the target y . in this way, the learned representations can be of good uniformity and sufficient predictive ability. unlike ib that minimizes input-representation information, we handle the information in an opposite way to avoid the overcompression issue of latent representations. besides, we design a conditional information minimization (cim) principle to mitigate the negative effect of potential redundant features from the input. in the information flow of @xmath1, infomax may introduces excessive and noisy information @xcite . for ifm, the task-irrelevant redundant nuisances features obtained by maximizing i(x; z) interfere with the optimization of maximizing i(y ; z). there are spurious correlations among these redundant features, forcing the model to learn a biased representation z. as a conditional regularization term for ifm, the cim principle eliminates negative redundant features while preserves noise-invariant features from inputs. under the ifm principle with the conditional regularization, cifm can extract noise-invariant sufficient representations for the input and target. we conduct experiments on 13 natural language understanding benchmarks. the results demonstrate that cifm can significantly improve the performance of plms for classification and regression. our cifm framework consistently achieves the best average performance compared to other methods, including 4 universal models and 7 representative deep representation learning technologies under different backbone models. for instance, with the roberta backbone, cifm improves average performance by +3.8% and +1.9% for classification and regression tasks compared to ce/mse, respectively. extended experiments prove that cifm can enhance the model's generalization including out-of-distribution and data-constrained scenarios, robustness to random and adversarial noise, and transferability to new tasks. and the results also indicates that the learned representations by cifm are more sufficient, robust and transferable. the contributions are as follows: 1) we propose an information flow maximization principle to learn more sufficient representations for the input and target by simultaneously maximizing both input-representation and representation-label information. 2) we design a conditional information minimization principle to eliminate negative redundant features while preserve noise-invariant features from the input. 3) we present an informationtheoretic cifm framework to learn noise-invariant sufficient representations for the input and target. it can enhance the generalization of plms for better language understanding. 4) experiments on 13 language understanding benchmarks demonstrate that cifm achieves better performance under different backbone models. extensive experiments show that the learned representations are more sufficient, robust and transferable. foot_1 ## methodology this section presents a new information-theoretic representation learning framework, named conditional information flow maximization (cifm), to extract sufficient representations for the input data and target task, as well as eliminates negative redundant features from the input. it contains two principles including information flow maximization and conditional information minimization. ## conclusion we propose an information-theoretic cifm framework to extract noise-invariant sufficient representations for the input data and target task. firstly, an ifm principle is proposed to learn more sufficient representations for the input and target. besides, a cim principle is designed to eliminate negative redundant features while preserve noise-invariant features from the input. experiments on 13 language understanding benchmarks demonstrate that cifm effectively improves the performance of plms for classification and regression. extensive experiments show that the learned representations are more sufficient, robust and transferable.