Scalable Infomin Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
The task of infomin learning aims to learn a representation with high utility whilebeing uninformative about a specified target, with the latter achieved by minimising the mutual information between the representation and the target.
It has broadapplications, ranging from training fair prediction models against protected attributes, to unsupervised learning with disentangled representations.
Recent workson infomin learning mainly use adversarial training, which involves training a neural network to estimate mutual information or its proxy and thus is slow and difficult to optimise.
Drawing on recent advances in slicing techniques, we propose anew infomin learning approach, which uses a novel proxy metric to mutual information.
We further derive an accurate and analytically computable approximationto this proxy metric, thereby removing the need of constructing neural networkbased mutual information estimators.
Compared to baselines, experiments on independence tests, disentangled representation learning and algorithmic fairnesstasks demonstrates that our new method has better utility and higher scalability.
Introduction
Learning representations that are uninformative about some target but still useful for downstreamapplications is an important task in machine learning with many applications in areas includingalgorithmic fairness [1, 2, 3, 4], disentangled representation learning [5, 6, 7, 8], information bottleneck [9, 10], and invariant representation learning [11, 12, 13].
A popular method for the above task is adversarial training [1, 3, 4, 7, 14, 2, 11], where two neuralnetworks, namely the encoder and the adversary, are trained jointly to compete with each other.
Theencoder’s goal is to learn a representation with high utility but contains no information about thetarget.
The adversary, on the contrary, tries to recover information about the target from the learnedrepresentation as much as possible.
This leads to a minmax game similar to that in generativeadversarial networks [15].
Adversarial training is effective with a strong adversary, however, itis often challenging to train the adversary thoroughly in practice, due to time constraints and/oroptimisation difficulties [16, 17].
In fact, recent studies have revealed that adversarial approachesmay not faithfully produce an infomin representation [18, 19, 17, 20, 21].
This motivates us to seeka good, adversarial training-free alternative for scalable infomin learning.
In this work, we propose a new method for infomin learning which is almost as powerful as usinga strong adversary but is highly scalable.
Our method is inspired by recent advances in informationtheory which proposes to estimate mutual information in the sliced space [22].
We highlight thefollowing contributions:
• We show that for infomin learning, a general, accurate estimate of mutual information or its boundis unnecessary: testing and minimising dependence randomly in some sliced space is sufficient;•
We develop an analytical approximation to the aforementioned sliced independence test, alongwith a scalable training algorithm based on this approximation.
No adversarial training is needed.
Importantly, the proposed method can be applied to a wide range of infomin learning tasks withoutany constraints on the form of variables or any assumption about the distributions.
This contrasts ourmethod to other adversarial training-free methods which are either tailored for discrete or univariatevariables [23, 24, 24, 25, 21] or rely on variational approximation to densities [10, 26, 18, 12].
Background
Infomin representation learning .
Let X ∈ R D be the data,
Y ∈ R D ′ be the target we want topredict from X .
The task we consider here is to learn some representation Z = f
( X ) that is usefulfor predicting Y but is uninformative about some target T ∈ R d .
Formally, this can be written as
[Equation]
where f is an encoder, L is some loss function quantifying the utility of Z for predicting Y and
I ( f ( X ); T ) quantifies the amount of information left in Z about T .
β controls the trade-off betweenutility and uninformativeness.
Many tasks in machine learning can be seen as special cases of thisobjective.
For example, by setting T to be (a set of) sensitive attributes e.g. race, gender or age, wearrive at fair representation learning [27, 1, 2, 20].
When using a stochastic encoder, by setting Y tobe X and T to be some generative factors e.g., a class label, we arrive at disentangled representationlearning [5, 6, 7, 8].
Similarly, the information bottleneck method [9, 10] corresponds to setting
T = X , which learns representations expressive for predicting Y while being compressive about X .
Adversarial training for infomin learning .
A key ingredient in objective (1) is to quantify
I ( f ( X ); T ) as the informativeness between f ( X ) and T .
One solution is to train a predictor for
T from f ( X ) and use the prediction error as a measure of I ( f ( X ); T ) [1, 3, 4].
Another approachis to first train a classifier to distinguish between samples from p ( Z, T ) vs. p ( Z ) p ( T ) [7, 14] or todistinguish samples Z ∼ p ( Z | T ) with different T [2, 11], then use the classification error to quantify
I ( f ( X ); T ) .
All these methods involve the training of a neural network t to provide a lower-boundestimate to I ( f ( X ); T ) , yielding a minmax optimisation problem
[Equation]
where ˆ I t ( f ( X ); T ) is an estimator constructed using t that lower-bounds I ( f ( X ); T ) .
The timecomplexity of optimising (2) is O
( L 1 L 2 ) where L 1 and L 2 are the number of gradient steps for theouter and inner loop respectively.
The strength of t is crucial for the quality of the learned representation [18, 19, 17, 20, 21].
For a strong adversary, a large L 2 is possibly needed, but this means along training time.
Conversely, a weak adversary may not produce a truly infomin representation.
Methodology
We propose an alternative to adversarial training for optimising (1).
Our idea is to learn representation by the following objective, which replaces I ( f ( X ); T ) in objective (1) with its ‘sliced’ version:
[Equation]
where SI denotes the sliced mutual information, which was also considered in [22].
Informally, SIis a ‘facet’ of mutual information that is much easier to estimate (ideally has closed form) but canstill to some extent reflect the dependence between Z and T .
Optimising (3) is then equivalent totesting and minimising the dependence between Z and T from one facet.
Importantly, while testingdependence through only a single facet may be insufficient, by testing and minimising dependencethrough various facets across a large number of mini-batches we eventually see I ( Z ; T ) → 0 .
We show one instance for realising SI whose empirical approximation ˆ SI has an analytic expression.
The core of our method is Theorem 1, which is inspired by [4, 22].
Theorem 1.
Let Z ∈ R D and T ∈ R
d are two random variables that have moments.
Z and T arestatistically independent if and only if SI ( Z, T ) = 0 where SI ( Z, T ) is defined as follows
[Equation]
(4) where ρ is the Pearson correlation, h, g : R → R are Borel-measurable non-constant functions, andθ ∈ S D − 1 ,
ϕ ∈ S
d − 1 are vectors on the surfaces on D -dimensional and d -dimensional hyperspheres.
Proof .
See the Appendix.
We sketch here how our result relates to [4, 22].
[22] considers SI ( Z ; T ) , defined as the expected mutual information E
[ I ( θ ⊤ Z, ϕ ⊤ T )] of slices of Z and T , where the expectation is takenover respective Haar measures θ ∈ S D − 1 , ϕ ∈ S d − 1 .
Proposition 1 in [22] shows that: (1) SI ( Z ; T ) ≥ 0 with equality iff Z and T are independent; and (2) SI ( Z ; T ) ≤ sup θ,ϕ I ( θ ⊤ Z, ϕ ⊤ T ) .
Our result uses some similar proof techniques, but instead of considering the mutual information I ( θ ⊤ Z, ϕ ⊤ T ) , we handle the Pearson correlation of the sup over functions h, g as defined above,which links to R´enyi’s maximal correlation [4, 28, 29, 30] and has closed form as shown below.
We call θ and ϕ the slices for Z and T respectively, and θ ⊤ Z , ϕ ⊤ T the sliced Z and T respectively.
Intuitively, Theorem 1 says that in order to achieve I ( Z ; T ) → 0 , we need not estimate I ( Z ; T ) inthe original space; rather we can test (and optimise) independence in the sliced space as realised by(4).
Note other realisations of the sliced mutual information SI can also be used.
The major merit ofthe realisation (4) is it has an analytic expression for its empirical approximation, as shown below.
Analytic approximation to SI .
An empirical approximation to (4) is
[Equation]
where
[Equation]
i.e., we approximate (4) by randomly sampling a number of slices θ, ϕ uniformly from the surfaceof two hyperspheres S D − 1 and S d − 1 and pick those slices where the sliced Z and the sliced Tare maximally associated.
With a large number of slices, it is expected that (5) will approximate (4)well.
We refer to [22] for a theoretical analysis on the number of required slices in estimator-agnosticsettings.
In Appendix B we also investigate empirically how this number will affect performance.
For each slicing direction, we further assume that the supreme functions h i , g j : R → R for thatdirection can be well approximated by K -order polynomials, i.e.
[Equation]
where σ ( · ) is a monotonic function which maps the input to the range of [ − 1 , 1] .
Its role is to ensurethat σ ( a ) always has finite moments, so that the polynomial approximation is well-behaved.
Note noinformation is lost by applying σ ( · ) .
Here we take σ ( · ) as the tanh function.
Other approximationscan also be used (e.g. those based on random features [31]), which can be explored in the future.
One may wonder if ignoring the higher-order terms σ ( a ) k ′ , k ′ >
K will cause inaccuracy in independence testing.
In Appendix A we show that if | ˆ h i ( a ) − h
i ( a ) | ≤ ϵ and | ˆ g j ( a ) − g j ( a ) | ≤ ϵ for∀ i, j, a , then as ϵ →
we have | ρ ∗ ij − ˆ ρ ∗ ij |
=
o ( ϵ ) where ρ ∗ ij =
ρ ( h i ( θ ⊤ i Z ) , g j ( ϕ ⊤ j T )) is the truecorrelation and ˆ ρ ∗ ij is its K -order approximation.
In such case we have ˆ ρ ∗ ij ≈
whenever ρ ∗ ij ≈ 0 .
With this polynomial approximation, the solving of each h i , g j reduces to finding their weights:
[Equation]
[Equation]
This is known as canonical correlation analysis [32] and can be solved analytically by eigendecomposition.
Hence we can find the weights for all pairs of h i , g j by doing S 2 eigendecompositions.
In fact, the functions h i , g j for all i, j can be solved simultaneously by performing a larger eigendecomposition only once.
We do this by finding w, v that maximise the following quantity:
[Equation]
where
[Equation] [Equation]
The benefits for solving f i , g j for all slices jointly are two-fold. The first is stronger test power. More specifically, it allows us to unify the power of different slices: while some sliced directions may be weak for detecting dependence, they together as a whole (by treating them as a ‘big slice’)can compensate for each other, yielding a powerful test.
We highlight that this joint-solving strategywill not violate our original objective (i.e. minimising (5)), as backed up by Theorem 2 below.
Theintuition is that if all Z ′ i and T ′ j together cannot achieve a high correlation, they alone can not either:
Theorem 2.
Provided that each h i , g j in (5) are K -order polynomials , given the sampled Θ ={ θ i } Si =1 , Φ = { ϕ j } Sj =1 , we have ˆ SI Θ , Φ ( Z ; T ) ≤ ϵ ⇒ sup
i,j sup h i ,g j ρ ( h i ( θ ⊤ i Z ) , g j ( ϕ ⊤ j T ))
≤ ϵ .
Proof .
See Appendix A.
Theorem 2 says that the solution of (6) yields an upper bound of (5).
This essentially means that weare safe to replace (5) with (6) which may be a more powerful independence test.
Another potentialbenefit for solving g j , h j jointly is its better efficiency in practice, as it avoids invoking a for loopand has better affinity to modern deep learning infrastructure and libraries which are optimised formatrix-based operations.
For these reasons, we use ˆ SI Θ , Φ ( Z ; T ) as an approximation to SI ( Z ; T ) .
Algorithm
Input: data D = { X ( n ) , Y ( n ) , T ( n ) } N n =
Output: Z = f
( X ) that optimises (1)
Hyperparams: β , N ′ , L 1 , L
Parameters: encoder f , MI estimator t for l 1 in 1 to L 1 do sample mini-batch B from D sample D ′ from D whose size N ′ < N // Max-step for l 2 in 1 to L 2 do t ← t + η ∇ t ˆ I t ( f ( X ); T ) with data in D ′ end for // Min-step f ← f − η ∇ f [ L ( f ( X ); Y )
+
β
ˆ
I t ( f ( X ); T ) ] with data in B end for return Z = f ( X )
Input: data D = { X ( n ) , Y ( n ) , T ( n ) } N n =
Output: Z = f
( X ) that optimises (1)
Hyperparams: β , N ′ , L , S
Parameters: encoder f , weights w, v in ˆ SI for l in 1 to L do sample mini-batch B from D sample D ′ from D whose size N ′ <
N // Max-step sample S slices Θ
= { θ i } Si =1 , Φ = { ϕ j } Sj
= solve the parameters w, v in ˆ SI analytically with Θ , Φ , D ′ by eigendecomposition //
Min-step f ← f − η ∇ f [ L ( f ( X ); Y )
+ β ˆ SI ( f ( X ); T ) ] with data in B end for return Z = f
( X ) Mini-batch learning algorithm . Given the above approximation (6) to SI , we can now elaboratethe details of mini-batch learning.
For each mini-batch B , we execute the following steps:
• Max-step .
Sample S slices Θ = { θ i } Si =1 , Φ = { ϕ j } Sj =1 and a subset of the data D ′ ⊂ D (here |D ′ | can be larger than |B| ).
Learn the weights w, v of (6) with Θ , Φ , D ′ by eigendecomposition;
• Min-step .
Set ˆ SI ( Z, T ) =
ρ ( w ⊤ Z ′ , v ⊤ T ′ ) with w, v solved in the max-step and Z ′ , T ′ defined in(6).
Update f by SGD: f ← f − η ∇ f [ L ( f ( X ) , Y ) +
β ˆ SI ( f ( X ) , T )] with the mini-batch data.
The whole learning procedure is shown in Algorithm 2. Compared to that of adversarial methods[2, 11, 1, 3, 20, 4] as shown in Algorithm 1, we replace the optimisation of neural net in the max stepwith an analytical eigendecomposition step.
The time complexity for eigendecomposition is O ( S 3 ) . As an optional strategy, one may actively seek informative slices for dependence detection. This canbe done by optimising ˆ SI w.r.t the sampled slices by a few gradient steps (e.g. 2-4) in the max-step: [Equation]
We can do this after solving ˆ SI analytically. Such a strategy is particularly useful when most ofthe sampled slices are weak in detecting dependence, which typically happens in late iterations oflearning where I ( Z ; T ) ≈ 0 . Since the optimisation of slices only happens in late iterations, andthat we only apply a small number of gradient steps, the increased execution time will be negligible.  Related works
Neural mutual information estimators .
A set of neural network-based methods [33, 34, 35, 36]have been proposed to estimate the mutual information (MI) between two random variable, most ofwhich work by maximising a lower bound of MI [37].
These neural MI estimators are in generalmore powerful than non-parametric methods [38, 39, 40, 41] when trained thoroughly, yet the timespent on training may become the computational bottleneck when applied to infomin learning.
Upper bound for mutual information .
Another line of method for realising the goal of infominlearning without adversarial training is to find an upper bound for mutual information [10, 18, 12,42].
However, unlike lower bound estimate, upper bound often requires knowledge of either theconditional densities or the marginal densities [37] which are generally not available in practice.
As such, most of these methods introduce a variational approximation to these densities whosechoice/estimate may be difficult.
Our method on the contrary needs not to approximate any densities.
Slicing techniques .
A series of successes have been witnessed for the use of slicing methods inmachine learning and statistics [43], with applications in generative modelling [44, 45, 46, 47],statistical test [48] and mutual information estimate [22].
Among them, the work [22] who proposesthe concept of sliced mutual information is very related to this work and directly inspires our method.
Our contribution is a novel realisation of sliced mutual information suitable for infomin learning.
Fair machine learning .
One application of our method is to encourage the fairness of a predictor.
Much efforts have been devoted for the same purpose, however most of the existing methods caneither only work at the classifier level [24, 23, 4, 49], or only focus on the case where the sensitiveattribute is discrete or univariate [21, 50, 25, 49, 27], or require adversarial training [2, 11, 1, 3, 20, 4].
Our method on the contrary has no restriction on the form of the sensitive attribute, can be used inboth representation level and classifier level, and require no adversarial training of neural networks.
Disentangled representation learning .
Most of the methods in this field work by penalising thediscrepancy between the joint distribution P = q ( Z ) and the product of marginals Q = (cid:81)
Dd q ( Z d )[6, 51, 26, 52, 7] 1 .
However, such discrepancy is often non-trivial to estimate, so one has to resortto Monte Carlo estimate ( β -TCVAE [26]), to train a neural network estimator (FactorVAE [7]) or toassess the discrepancy between P and Q by only their moments (DIP-VAE [51]).
Our method avoidsassessing distribution discrepancy directly and instead perform independence test in the sliced space.
Experiments
We evaluate our approach on three tasks: independence testing, fairness machine learning and disentangled representation learning.
Code for reproducing our results is in the supplementary material.
Evaluation metric .
To assess how much information is left in the learned representation Z ∈ R Dabout the target T ∈ R K , we calculate the maximal R ´ enyi correlation ρ ∗ ( Z, T ) between Z and T :
[Equation]
(8) which has the properties ρ ∗ ( Z, T )
=
if and only if Z ⊥ T and ρ ∗ ( Z, T )
= 1 if h ( Z ) =
g ( T )for some deterministic functions h, g [28].
One can also understand this metric as the easiness ofpredicting (the transformed) T from Z , or vice versa.
As there is no analytic solution for the supremum in (8), we approximate them by two neural networks h, g trained with SGD.
Early stoppingand dropout are applied to avoid overfitting.
The reliability of this neural approximation has beenverified by the literature [4] and is also confirmed by our experiments; see Appendix B.
This metric is closely related to existing metrics used in fairness and disentangled representationlearning such as demographic parity (DP) [23] and total correlation (TC) [7].
For example, ifρ ∗ ( Z, T ) → 0 then it is guaranteed that ˆ Y ⊥ T for any predictor ˆ Y = F ( Z ) , so ρ ∗ ( Z, T ) isan upper bound for DP .
Similarly, ρ ∗ ( Z, T ) coincides with TC which also assesses whether Z ⊥ T .
Baselines .
We compare the proposed method (denoted as “Slice”) with the following approaches
[Figure] • Pearson , which quantifies I ( Z ; T ) by the Pearson correlation coefficient 1 DK
[Figure]
It was used in [5, 51] to learn disentangled representations; [Figure] • dCorr , a non-parametric method for assessing independence based on the correlation between the [Figure]
pairwise distances [41].
It was applied in [36] as a surrogate to MI for representation learning; [Figure]
• Neural R ´ enyi , an adversarial method for fair machine learning [24] which quantifies I ( Z ; T ) bythe R ´ enyi correlation ρ ∗ ( Z, T ) =
sup h,g ρ ( h ( Z ) , g ( T )) with h, g approximated by neural networks.
Many existing methods [1, 2, 3, 4] can be seen as variant of this approach;• Neural TC , an adversarial method for learning disentangled representation [7, 56] which quantifiesI ( Z ; T ) by the total correlation
TC ( Z, T ) =
KL [ p ( Z, T )
∥
p ( Z ) p ( T )] .
To computes TC, a (neuralnetwork) classifier is trained to classify samples from p ( Z, T ) and samples from p ( Z ) p ( T ) .
Manyexisting methods [57, 1] can be show to some extent t equivalent to this method (see Appendix B).• VB , i.e. variational upper bound, which introduces a (learnable) variational distribution q ( T | Z ) toform an upper bound of MI [42]: I ( Z ; T ) ≤ E p ( Z,T )
[log q ( T | Z )] − E p ( Z ) p ( T ) [log q ( T | Z )] .For
a fair comparison, for adversarial training-based approaches (i.e. Neural R ´ enyi, Neural TC),we ensure that the training time of the neural networks in these method are at least the same as theexecution time of our method or longer.
We do this by controlling the adversarial steps L 2 in (2) and
Algorithm 1.
The same setup is used for VB. See each experiment for the detailed execution time.
Hyperparameter settings .
throughout our experiments, we use 200 slices.
We find that this settingis robust across different tasks.
An ablation study on the number of slices is given in Appendix B.
The number of samples used to learn the parameter of w, v in (6) is 10,000 sample and is kept fixed.
The order of the polynomial used in (6) namely K is set as K = 3 and is fixed across different tasks.
Computational resource .
All experiments are done with a single Nvidia GeForce Tesla T4 GPU.
5.1 Independence testing
We first verify the efficacy of our method as a light-weight but powerful independence test.
Forthis purpose, we investigate the test power of the proposed method on various synthetic dataset withdifferent association patterns between two random variables X ∈ R 10 , Y ∈ R 10 and comparedto that of the baselines.
The test power is defined as the ability to discern samples from the jointdistribution p ( X, Y ) and samples from the product of marginal p ( X ) p ( Y ) and is expressed as aprobability p ∈ [0 , 1] .
The data is generated as Y =
(1 − α ) ⟨ t ( A X ) ⟩ + αϵ, X d ∼ U [ − 3 , 3] , A dd =1 , A dk = 0 .
2 , ϵ ∼ N ( ϵ ; 0 , I ) , α ∈ (0 , 1) and ⟨·⟩ is a scaling operation that scales the operand tothe range of [0 , 1] according to the minimum and maximum values in the population.
The functiont ( · ) determines the association pattern between X and Y and is chosen from one of the following:t ( a )
= a, a 2 , sin( a ) , tanh ( a ) .
The factor α controls the strength of dependence between X and Y .
All tests are done on 250 samples and are repeated 1,000 times.
We choose this sample number asit is a typical batch size in mini-batch learning.
For methods involving the learning of parameters(i.e.
Slice, Neural R´enyi, Neural TC), we learn their parameters from 10,000 samples.
The time forlearning the parameters of Slice, Neural R´enyi and Neural TC are 0.14s, 14.37s, 30.18s respectively.
For completeness, we also compare with the ‘optimal test’ which calculates the R´enyi correlationρ ∗ ( X, Y ) =
ρ ( h ( X ) , g ( Y )) with the functions h, g exactly the same as the data generating process.
[Table] Figure 1 shows the power of different methods under various association patterns t and dependencelevels α .
Overall, we see that the proposed method can effectively detect dependence in all cases, andhas a test power comparable to neural network-based methods.
Non-parametric tests, by contrast,fail to detect dependence in quadratic and periodic cases, possibly due to insufficient power.
Neural
TC is the most powerful test among all the methods considered, yet it requires the longest time totrain.
We also see that the proposed method is relatively less powerful when α ≥ 0 .
8 , but in suchcases the mutual information between X and Y is indeed very low (see Appendix B).
The resultssuggest that our slice method can provide effective training signals for the task of infomin learning.
5.2 Fairness
For this task, we aim to learn fair representations Z ∈ R 80 that are minimally informative aboutsome sensitive attribute T .
We quantify how sensitive Z is w.r.t T by R ´ enyi correlation ρ ∗ ( Z, T )calculated using two neural nets.
Smaller ρ ∗ ( Z, T ) is better.
The utility of the learned representationi.e.
, L ( Z ; Y ) is quantified by ρ ∗ ( Z, Y ) .
This formulation for utility, as aforementioned, is equivalentto measuring how well we can predict Y from Z .
In summary, the learning objective is:
[Equation]
where I ( Z ; T ) is estimated by the methods in test or replaced by a proxy.
For each dataset considered, we use 20,000 data for training and 5,000 data for testing respectively.
We carefully tune thehyperparameter β for each method so that the utility ρ ∗ ( Z ; Y ) in that method is close to that in theplain model (i.e. the model trained with β = 0 , denoted as N/A below.
Other experiments belowhave the same setup).
All figures are reported on the test set.
US Census Demographic data .
This dataset is an extraction of the 2015 American Community
Survey, with 37 features about 74,000 census tracts.
The target Y to predict is the percentage ofchildren below poverty line in a tract, and the sensitive attribute T is the ratio of women in that tract.
The result is shown in Table 1.
From the table we see that the proposed slice method produces highlyfair representation with good utility.
The low ρ ∗ ( Z, T ) value indicates that it is unlikely to predict Tfrom Z in our method.
While adversarial methods can also to some extent achieve fairness, it is stillnot comparable to our method, possibly because the allocated training time is insufficient (we verifythis in Appendix B).
Non-parameteric methods can not produce truly fair representation, despitethey are fast to execute.
VB, which minimises the upper bound of MI, achieves better fairness thanadversarial methods on average, but has a higher variance, possibly due to the reason shown in [58].
UCI Adult data .
This dataset contains census data for 48,842 instances, with 14 attributes describing their education background, age, race, marital status, etc.
Here, the target Y to predict is whetherthe income of an instance is higher than 50,000 USD, and the sensitive attribute T is the race. [Table] [Figure] [Figure]
The result is summarised in Table 2.
Again, we see that the proposed slice method outperformsother methods in terms of both fairness and utility.
For this dataset, Neural R ´ enyi also achieves goodfairness, although the gap to our method is still large.
Neural TC, by contrast, can not achieve acomparable level of fairness under the time budget given; a phenomenon also observed in the US
Census dataset.
This is possibly because the networks in Neural TC require longer time to train.
VB-based method does not work very satisfactorily on this task, possibly because the allocated timeto learn the variational distribution q ( Z | T ) is not enough, leading to a loose upper bound of I ( Z ; T ) .
5.3 Disentangled representation learning
We next apply our method to the task of disentangled representation learning, where we wish todiscover some latent generative factors irrelevant to the class label T .
Here, we train a conditionalautoencoder X ≈ G ( Z, T ) to learn representation Z = E ( X ) which encodes label-irrelevant information of X .
The target to recover is Y = X .
The utility of Z is therefore quantified as thereconstruction error:
L ( Z ; Y ) =
E [ ∥ G ( Z, T ) − X ∥ 22 ] , resulting in the following learning objective:
[Equation]
The conditional autoencoder uses a architecture similar to that of a convolutional GAN [59], withthe difference being that we insert an adaption layer
Z ′ =
MLP ( Z, T ) before feeding the featuresto the decoder.
See Appendix B for the details of its architecture.
All images are resized to 32 × 32.
Dsprite .
A 2D shape dataset [60] where each image is generated by 4 latent factors: shape, rotation, scale, locations.
Here the class label T is the shape, which ranges from (square, ellipse,heart).
For this dataset, we train the autoencoder with 100 iterations with a batch size of 512.
Thedimensionality of the representation for this task is 20 i.e. Z ∈ R 20 .
As in the previous experiments, we provide quantitative comparisons of the utility and disentanglement of different methods in Table 3.
In addition, we provide a qualitative comparison in Figure
[Table]
[Table]
2 which visualises the original image, the reconstructed image and the reconstructed image with aswapped label.
From Table 3, we see that the proposed method achieve very low ρ ∗ ( Z, T ) whilemaintaining good MSE, suggesting that we may have discovered the true label-irrelevant generativefactor for this dataset.
This is confirmed visually by Figure 2(b), where by changing T in reconstruction we only change the style.
The separation between T and Z is less evident in adversarialapproach, as can be seen from Table 3 as well as Figure 2(a) (see e.g. the reconstructed ellipses inthe third column.
They are more like a interpolation between ellipses and squares).
For this task,
VB method indeed works quite well, being the second best performing method on this task.
CMU-PIE .
A colored face image dataset [61] where each face image has different pose, illumination and expression.
We here consider a cropped version of it [62].
Here the class label T is theexpression, which ranges from (neutral, smile, blinking).
We train an autoencoder with 200 iterationand a batch size of 128.
The dimensionality of the representation for this task is 128 i.e. Z ∈ R 128 .
Figure 3 and Table 4 shows the qualitative and quantitative results respectively.
From Figure 3, wesee that our method can well disentangle expression and non-expression representations: one caneasily modify the expression of an image by only changing T in reconstruction.
Other contents ofthe image e.g. the pose, illumination, identity are not changed.
Adversarial approach can also tosome extent achieve disentanglement between Z and T , however such disentanglement is imperfect:not all of the instances can change the expression by only modifying T .
This is also confirmed quantitatively by Table 4, where one can see the relatively high ρ ∗ ( Z, T ) values in adversarial methods.
VI method, which optimises the upper bound of MI, again achieves a good performance on this task,being much better than adversarial methods.
However, the proposed method still performs the best.
Conclusion
This work proposes a new method for infomin learning without adversarial training.
A major challenge is how to estimate mutual information accurately and efficiently, as MI is generally intractable.
We sidestep this challenge by only testing and minimising dependence in a sliced space, which canbe achieved analytically, and we showed is sufficient for our goal.
Experiments on algorithmic fairness and disentangled representation learning demonstrate the method’s effectiveness of our method.
Through our controlled experiments, we also verify that adversarial approaches indeed may notproduce truly infomin representation – an observation consistent with recent studies.
This suggeststhat existing adversarial approaches may not converge to good solutions, or may need more time forconvergence, with more gradient steps needed to train the adversary fully.
The result also hints at thepotential of diverse randomisation methods as an alternative to adversarial training in some cases.
While we believe our method can be used in many applications for societal benefit (e.g. for promoting fairness), since it is a general technique, one must always be careful to prevent societal harms.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seeintroduction and conclusion
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] see Appendix A3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] codes provided in supplemental material; all used dataset are public
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] see page 5 ’hyperparameter settings’
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] see page 5 ’computational resource’
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] all data are available in the public and
the only requirement in the license is not to claim the ownership of data
(c) Did you include any new assets either in the supplemental material or as a URL
[Yes]
[Table]
the new codes developed for reproducing the experiments
[Table] (d) Did you discuss whether and how consent was obtained from people whose data [Table]
you’re using/curating?
[No] all dataset used is available in the public [Table]
(e) Did you discuss whether the data you are using/curating contains personally identifi- [Table]
able information or offensive content?
[No] no sensitive information is in the dataset
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]