R EPAIRING S YSTEMATIC O UTLIERS BY L EARNING C LEAN S UBSPACES IN VAE
S
Anonymous authors
Paper under double-blind review
A BSTRACT
Data cleaning often comprises outlier detection and data repair.
Systematic errors result from nearly deterministic transformations that occur repeatedly in the data, e.g. speciﬁc image pixels being set to default values or watermarks.
Consequently, models with enough capacity easily overﬁt to these errors, making detection and repair difﬁcult.
Seeing as a systematic outlier is a combination of patterns of a clean instance and systematic error patterns, our main insight is that inliers can be modelled by a smaller representation (subspace) in a model than outliers.
By exploiting this, we propose Clean Subspace Variational Autoencoder (CLSVAE) , a novel semi-supervised model for detection and automated repair of systematic errors.
The main idea is to partition the latent space and model inlier and outlier patterns separately.
CLSVAE is effective with much less labelled data compared to previous related models, often with less than 2% of the data.
We provide experiments using three image datasets in scenarios with different levels of corruption and labelled set sizes, comparing to relevant baselines.
CLSVAE provides superior repairs without human intervention, e.g. with just 0.25% of labelled data we see a relative error decrease of 58% compared to the closest baseline.
I NTRODUCTION
Often practitioners have to deal with dirty datasets before they can start applying machine learning (ML) models.
We focus on datasets where some instances have been corrupted by noise, producing outliers.
The corresponding clean instances prior to corruption are called inliers, as well as any other instances that have not been corrupted.
Because the presence of outliers can degrade the performance of ML methods (Krishnan et al., 2016; Liu et al., 2020), a standard option is to resort to a data cleaning pipeline before applying any model.
This pipeline includes two key tasks: i) outlier detection (Ruff et al., 2021), detecting all outliers; ii) repair those outlier instances (Neutatz et al., 2021; Wan et al., 2020), recovering the underlying inlier instance.
Ultimately, the goal is to propose a method that performs these steps automatically, i.e. automatic detection and repair .
Generally, we can consider two types of errors present in an outlier: random or systematic (Taylor, 1997; Liu et al., 2020).
Random errors corrupt each instance independently, and feature value changes are sampled from an unknown distribution.
This noising cannot be replicated in a repeatable manner.
For continuous features, a common example of this type of error are those well-modelled by additive noise with zero-mean.
Systematic errors result from deterministic, or nearly deterministic, transformations that occur repeatedly in the data.
Examples of systematic errors include watermarks or deterministic pixel corruption (e.g. artifacts) in images; additive offsets or replacement by default values (e.g. NaN) in sensor data; deterministic change of categories (mislabelling) or of name formats in categorical features in tabular data.
Usually, the same features are affected, but not always.
In most cases, this noising can be replicated.
These two types of errors have a different impact when it comes to models performing detection or repair.
Random errors do not show a distinct pattern across outliers and thus are not predictable.
As a result, unsupervised models using regularization or data reweighting (Zhou & Paffenroth, 2017;
Akrami et al., 2019; Eduardo et al., 2020) can avoid overﬁtting to random errors.
On the contrary, a systematic error shows a pattern across outliers, as a result of the (nearly) deterministic transformation, making them predictable (Liu et al., 2020).
This property makes higher capacity models (e.g. deep learning) prone to overﬁtting to these errors, even in the presence of regularization.
As a re- sult, models for outlier detection and repair in the presence of systematic errors more easily conﬂate outliers with inliers.
In this work we study how to develop a model for data cleaning robust to the effect of systematic errors.
One solution is to provide some supervision, so the model can distinguish between inliers and outliers with systematic errors.
This supervision can be provided in different forms, such as logic rules (Rekatsinas et al., 2017) or programs (Lew et al., 2021) describing the underlying clean data.
However, these may require expert knowledge or substantial effort to formalize.
Hence, it is easier and less time consuming for the practitioner to simply provide a trusted set as form of supervision.
A trusted set is a small labelled subset of the data, which can be used to train a method for detection and repair.
The user labels the instances either inlier or outlier, providing a few examples (e.g.instances) per type of systematic error to repair.
Overall, this might correspond to less than 2% of the entire dataset ( sparse semi-supervision ).
Equally important, labelling does not require the user to manually repair the instances in the trusted set.
Deep generative models (DGM) have high capacity (ﬂexible) and thus can easily overﬁt to systematic errors.
This motivates us to propose a method for detection and repair of such errors based on a semi-supervised DGM, which to the best of our knowledge has not been explored.
We postulate that inlier data needs a smaller representation relative to outlier data when being represented by DGMs which we call the compression hypothesis .
In addition, we claim that outliers are a combination of a representation describing the inlier portion of an instance, and a representation describing the type of systematic error .
Given these insights, we propose Clean Subspace Variational Autoencoder (CLSVAE), a novel semisupervised model for detection and automated repair of systematic errors.
This model deviates from a standard VAE (Kingma & Welling, 2014), in two ways.
First, the latent representation is partitioned into two subspaces: one that describes the data if it were an inlier (clean subspace), and the other that describes what systematic error (if any) has been applied (dirty subspace).
The model is encouraged to learn a disentangled representation using a simple yet effective approach: for outliers the decoder will use a clean subspace concatenated with the dirty subspace; in turn, for inliers the decoder will reuse the same clean subspace but concatenated with random noise.
Then, at repair time the decoder will only need the clean subspace to reconstruct the underlying inlier.
Secondly, we introduce semi-supervision through a trusted set, in so helping the model distinguish between inliers and outliers with systematic errors.
Additionally, to encourage the clean subspace to represent inlier data, and the dirty subspace to represent systematic errors, we introduce a novel penalty term minimizing their mutual information (MI).
This penalty is based on the distance correlation (DC) (Sz ´ ekely et al., 2007), and it improves model performance (stability) and repair quality.
Compared to baselines we provide superior repairs, particularly, we show signiﬁcant advantage in smaller trusted sets or when more of the dataset is corrupted.
R ELATED W ORK
For random errors, several works have explored detection (Akrami et al., 2019; Ruff et al., 2019; Liu et al., 2020; Lai et al., 2019).
Some have proposed methods for detection and automated repair of random errors (Eduardo et al., 2020; Zhou & Paffenroth, 2017; Krishnan et al., 2016).
A few works have explored outlier detection for systematic outliers, both semi-supervised (Ruff et al., 2019) and unsupervised (Liu et al., 2020; Lai et al., 2020), but these works do not consider automated repair.
For tabular data, methods have been proposed that can do repair after detection has been performed; for an overview, see (Ilyas & Chu, 2019; Chu et al., 2016).
These use probabilistic models melded with logic rules, i.e. probabilistic relational models (Rekatsinas et al., 2017), or user-written programmatic descriptions of the data, e.g. in a probabilistic programming language (Lew et al., 2021).
They have the potential to capture systematic errors.
However, these require the user to provide rules or programs that characterize clean data, which requires manual effort and expertise.
In contrast, labelling a few inlier and outlier instances to build a trusted set may be more user-friendly.
The idea of using unsupervised models for outlier detection is not new (Sch¨olkopf et al., 1999; Liu et al., 2008).
Unsupervised removal from a dataset of a small fraction of instances suspected of being outliers has been explored in Koh et al.
; Diakonikolas et al.
Liu et al.
(2020), when used against adversarial errors it is called data sanitization (Koh et al., 2018).
However, if one wants to repair existing dirty data, a common unsupervised approach for moderate corruption is to apply enough regularization or data reweighting to an autoencoder (Eduardo et al., 2020; Zhou & Paffenroth, 2017; Akrami et al., 2019), hopefully repairing the errors at reconstruction.
This has proven successful for random errors, though as we show, this is less effective for systematic errors.
Often regularization is too strong leading to bad repair quality, since reconstruction is collapsing to mean behaviour – e.g. blurry image samples, missing details.
Attribute manipulation models (Klys et al., 2018; Choi et al., 2020) usually rely on extensive labelled data, usually fully supervised.
Data cleaning can be seen as an attribute manipulation problem with one attribute: either the instance is an inlier or outlier.
Some models like CVAE (Sohn et al., 2015) may use discrete latent variables, instead of continuous, to model attributes.
These may lack the capacity to capture diversity in the same attribute (Joy et al., 2020), e.g. distinct types of systematic errors in the data, and thus offer a poor ﬁt to the data.
Later, this may result in poor repair.
Disentanglement models , unsupervised (Locatello et al., 2019; Tonolini et al., 2019) and semisupervised (Ilse et al., 2020; Locatello et al., 2019; Joy et al., 2020), encourage individual (continuous) latent variables to capture different attributes of the data instances.
In theory, a disentanglement model could isolate the inlier / outlier attribute into a latent variable, then after training one could ﬁnd a value for this variable that repairs the outlier.
In practice, this is much simpler with semisupervised models, since we know which variable corresponds the attribute, but these tend to use more labelled data than we consider.
In either case, these models usually need additional processing, often requiring human in the loop to explore the latent variable and ﬁnd the best repair.
Conversely, once our model is trained, further human intervention is not needed, so that repair is automated.
P ROBLEM D EFINITION
We assume the user knows that systematic errors have corrupted dataset X , and thus outliers exist therein, but the inliers are still the majority.
The user also has an idea of what patterns constitute systematic errors, and is able to recognize them.
Intuitively, we think of inlier data being characterized by a set of patterns, which we call clean patterns , and those that constitute (systematic) errors, called dirty patterns .
An outlier is an instance that has been corrupted by systematic errors, where prior to that would be constituted by clean patterns only .
Formally, for ˜ x an underlying inlier instance, an outlier is deﬁned by x = f cr (˜ x ) .
We deﬁne f cr as a general transformation that corrupts the inlier instance with systematic errors , and not necessarily invertible.
Each type of systematic error usually affects speciﬁc features of ˜ x in the same predictable way, repeatedly in the dataset.
This is unlike random errors, which both the feature and changed values are at random throughout instances.
The user builds a small trusted set by labelling a few of the inliers and outliers in the train set X , forming the labelled subset X l = { x n } N l n =1 .
So we have the dirty dataset X =
X l ∪ X u , where
X u = { x n } N u n =1 is the unlabelled part.
The overall size of the train set is N = N l + N u .
Each x n ∈ X l is associated with a label y n ∈ { 0 , 1 } , which indicates whether x n is an inlier ( y n = 1 ) or an outlier ( y n = 0 ).
We write Y l = { y n } N l n =1 and thus the trusted set is formally deﬁned by ( X l , Y l ) .
The trusted set should be representative of the inliers and outliers in the data.
Note that there is a set of different corrupting transformations, i.e. systematic error types, each applied to several instances.
Hence, the trusted set should provide at least a few labelled samples per type of systematic error.
This is important so as to help the model distinguish between inliers and outliers.
In our problem, the labelled portion of the dataset X l (trusted set) is signiﬁcantly smaller than the unlabelled portion
X u , e.g. 0.5% of N u .
Given how small the trusted set is, we refer to this as sparse semi-supervision .
In this work, the main task is to ﬁrst perform outlier detection on X , in order to discover y for each instance x .
The second task is repair those instances that are considered to be outliers ( y = 0 ).
Speciﬁcally, the model needs to provide a repair transformation g r such that ˆ x = g r ( x )
where ˆ x ≈ ˜ x
(i.e. repair is close enough to the underlying inlier instance).
Thus, we want a model that performs these two steps automatically, i.e. without a human in the loop.
O UR P ROPOSAL : C LEAN S UBSPACE VAE (CLSVAE)
In this section, we introduce the generative and variational models for our proposal.
Our Variational
Autoencoder (VAE) model is motivated by three observations:
• Outliers are systematic .
Outlier instances can be described by a number of predictable recurring patterns, some of which are considered dirty (e.g. black patch on an image).
Our assumption is that the latter patterns can be well represented by a DGM.
• Compression hypothesis .
Inlier data can be compressed further than outlier data.
We assume that less capacity (parameters or variables) is needed by DGMs to represent inliers.
• Outliers are a combination of clean and dirty patterns .
Because clean and dirty patterns produce different visible effects in the instance, the effects of clean and dirty patterns can be modelled separately.
Hence, this DGM only requires the parameters (or variables) of the clean patterns to generate a repair.
We call our model the sparse semi-supervised Clean Subspace VAE (CLSVAE) .
This reﬂects the fact that our model has a partitioned latent space: a subspace for clean patterns, and a subspace for dirty patterns.
Thus at test time only the subspace for clean patterns is used to generate a repair.
A standard VAE (Kingma & Welling, 2014), with weight decay, is provided in Annex A for readers unfamiliar with this type of model.
4. G ENERATIVE M ODEL
The main idea is that if the compression hypothesis holds inlier samples ( y = 1 ) will have a smaller representation relative to outliers ( y = 0 ), whilst outliers need an extra representation to model errors.
In our model, z c will represent inliers, z d will represent the error pattern for outliers, and z (cid:15) is random noise.
The overall latent code is z =
[ z c ; y z (cid:15) + (1 − y )
z d ] , where [ ; ] deﬁnes vector concatenation, leading to a latent space that is partitioned into two subspaces.
If generating an inlier ( y = 1 ) then z c ∈ R q is by itself responsible for modelling all the clean patterns present in an instance, encouraging a lower dimensional manifold for inlier data.
This is due to z (cid:15) being a random noise vector of the same dimension as z d ∈ R p , devoid of additional information.
If generating an outlier ( y = 0 ) both z c and z d are used, where z d models the different types of dirty patterns that can be detected in an instance.
In fact, using z (cid:15) during inlier generation encourages z
d to model dirty patterns only, encouraging a higher dimensional manifold for outlier data.
Our generative model is therefore deﬁned by the joint distribution between y and latent subspaces z c and z d .
The main idea expressed above can be written as a two component mixture model for the decoder, see eq.
(4), where y is the gating variable.
Hence we have
[Equation]
where p
[Equation]
where the density p θ ( x | [ · ; · ]) is parameterized by a neural network.
We assume inlier data has a smaller variance as whole than outlier data, so we use a σ c that is much smaller than σ d .
Note that z (cid:15) is just Gaussian random noise 1 with σ (cid:15) < σ d .
Hence, after training, the region around(zero mean) for this subspace encourages p θ ( x | [ · ; · ]) to only use z c for reconstruction, obtaining a repaired instance.
The parameter α reﬂects the prior belief on the fraction of clean data.
Smaller values for α means more data points are rejected when modelling z c , which offers more robustness.
So we have σ (cid:15) , σ c , σ d and α as hyper-parameters.
4. V ARIATIONAL M ODEL
We consider separate encoders for z c and z d , and make y depend on z c and z d .
The idea is that parameters of each encoder focus on different aspects, i.e. clean or dirty patterns respectively.
The model factorizes as
[Equation]
and
[Equation] [Equation]
where for distributions in eq.
(6): { µ φ c ( . ) , σ φ c ( . ) } is a neural network with parameters φ c ; similarly for { µ φ d ( . ) , σ φ d ( . ) } with φ d .We have σ φ c ( x ) and σ φ d ( x ) being diagonal covariance matrices.
The distribution for random noise z (cid:15) is the same as in the generative model.
In eq.
(7), the π φ y ( . ) parametrizes the Bernoulli distribution q φ y ( y | z c , z d ) , and is a neural network with parameters φ y .We found that using q φ y ( y | z c , z d ) yielded better results than using q φ y ( y | z d ) or q φ y ( y | x ) .
Since z c provides important context on the clean patterns present, which in turn allows z d to better focus on modelling dirty patterns.
In practice, to stabilize the optimization procedure in a few cases, we used π φ y ([ sg ( z c ); z d ]) in q φ y ( y | z c , z d ) – eq.
Note that sg ( z c ) stands for stop gradient operator applied to z c ; and this prevents z c from being updated with dirty pattern information early in the training.
A more detailed discussion on q φ y ( y | z c , z d ) is presented in Annex E.
4. T RAINING L OSS
Our model is trained to maximize an objective function with three terms, which accounts for our semi-supervised setting.
The ﬁrst term L ( x ) is the evidence lower bound (ELBO) for the unlabelled part of the data.
The second term L ( x , y ) is the ELBO for the trusted set.
The third term L WCE ( x , y ) is the weighted cross-entropy loss which ensures that q φ y ( y | z c , z d ) correctly predicts the trusted set labels y .
The ELBO for the unlabelled (unsupervised) part is
[Equation]
where q σ (cid:15) ( z (cid:15) )
=
p σ (cid:15) ( z (cid:15) ) and so they cancel each other.
The expectations are obtained via MonteCarlo (MC) estimation via reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014).
For the trusted set (supervised) part the ELBO is
[Equation]
Lastly, we need to deﬁne the weighted cross-entropy L WCE ( x , y ) , which is
[Equation]
where
[Equation]
and ω imb compensates for trusted set class imbalance.
However, we do not have access to q ( y = 1 | x ) and thus we cannot estimate L WCE ( x , y ) directly.
Still, we can minimize an upper-bound
[Equation]
which is obtained by applying Jensen’s inequality.
Combining the three terms deﬁned above, we minimize the overall loss
[Equation]
with respect to the generative and variational parameters.
The hyperparameter β value controls the amount of up-sampling and importance relative to the other terms, which tends to be moderately high due to how small the trusted set is.
4. D ISTANCE C ORRELATION P ENALTY
Ideally, z c captures clean patterns only, whilst z d captures dirty patterns.
Therefore z c and z d should have low mutual information (MI).
However, in more challenging scenarios, e.g. small trusted set or higher dataset corruption, obtaining this solution may not be guaranteed.
Enforcing a constraint encouraging low MI between z c and z d will lead to better model performance and stability in challenging scenarios, improving repair quality.
We would like to introduce a constraint that minimizes MI between z c and z d .
However, approximating MI properly can be complex.
Instead, we use distance correlation (DC) as a surrogate for
MI (Sz´ekely et al., 2007), which is easier to compute and can measure non-linear dependencies between vector variables.
Other works have used DC as a surrogate for MI, e.g. (Chen et al., 2021).
Further, DC can also measure dependence between vector variables of different dimensions, which is often the case with z c and z d .
For the data batch ( z c , z d ) ∈
( Z c , Z d ) where Z c ∈ R N × q and
Z d ∈ R N × p , we can deﬁne the empirical estimate of DC as dCorr N ( Z c , Z d ) .
The deﬁnition of the estimator dCorr N ( Z c , Z d ) can be seen in Annex F.
Essentially, DC is the standard correlation between the elements of the double centered pairwise distance matrices of each data batch Z c and Z d .
The range is 0 ≤ dCorr N ( Z c , Z d ) ≤ 1 , where 0 means variables are independent, and 1 implies that z c and z d are strongly correlated.
Enforcing this constraint means adding a penalty to the model training loss.
Hence, reusing the loss deﬁned in eq.
(13) we now have
[Equation]
(14) where λ t increases every epoch from 0 until it reaches a maximum value
λ T , which is then maintained.
The rate of increase of λ t and λ T are hyper-parameters.
This strategy is a type of penalty method as used in constrained optimization.
4. O UTLIER D ETECTION AND R EPAIR P ROCESS
After training, we proceed with outlier detection and automated repair , as in Section 3.
The detection task is to discover the ground-truth labels y for each x ∈ X , using inferred labels ˆ y .
A score
A ( x ) and threshold γ ≥ 0 are used to get the set of outliers O =
{ x ∈ X|
A ( x ) ≥ γ } , where a higher A ( x ) means the more likely x is an outlier.
The inferred label ˆ y is obtained as: (inlier)
ˆ y
=if x / ∈ O ; (outlier)
ˆ y
=
if x ∈ O .
For our model, we use a score based on the negative log probability of inlier given the latent subspaces, which is
[Equation]
The threshold γ can be chosen as γ ≈ − log(0 .
5) assuming q φ y ( y = 1 | . )
=
π φ y ( . ) is near calibrated, or it can be user-deﬁned.
The repair task is to obtain an inferred reconstruction ˆ x from the outlier x ∈ O such that it is close to the inlier ground-truth ˜ x .
The repair is generated using the most likely reconstruction under our model for y = 1 (inliers), which means only the clean subspace z c is used.
This is the maximum a posteriori estimate for a VAE, where one approximates p θ ( z c | x ) by q φ ( z c | x ) , and then uses the means of q φ ( z c | x ) , p (cid:15) ( z (cid:15) ) and p θ ( x | [ z c ; z (cid:15) ]) in the estimate.
Hence, we have
[Equation]
E XPERIMENTS
We evaluate two tasks: outlier detection , and automated repair .
Our experiments use three image datasets: Frey-Faces 3 , Fashion-MNIST (Xiao et al., 2017), Synthetic-Shapes .
Synthetic-Shapes is a synthetic dataset built around four different shapes (classes): a circle, a rectangle, an ellipse and a triangle.
These are colored white and set in a black background.
We corrupt datasets with synthetic systematic errors, since public real-world datasets with ground-truth repairs and respective labels are difﬁcult to ﬁnd, as seen in Eduardo et al.
; Krishnan et al.
; Liu et al.
We compare our model (CLSVAE) with baselines ranging from completely supervised to unsupervised.
acceptance.
[Table]
Evaluation For outlier detection we use AVPR (Average Precision) (Everingham et al., 2015) to measure detection quality, which is a surrogate for the area under the precision-recall curve (Hendrycks & Gimpel, 2016).
This metric is preferred since it is insensitive to label imbalance, typical in outlier detection.
AVPR score is between [0 , 1] and higher means better.
For automated repair we want to quantify the quality of the repair, for outlier instances.
We report the standardized mean squared error (SMSE) between pixels of the ground-truth (inlier) instance and that of the proposed repair.
We report SMSE separately for the dirty pixels (those affected by the systematic error) and for the clean pixels (those unaffected).
The ﬁrst measures repair performance, while the latter measures distortion that the repair process causes to clean pixels.
In both cases, a lower SMSE means better.
Note in the case of binary pixels, the SMSE is just the Brier score, and thus is in [0 , 1] .
Datasets and Corruption Process
In our experiments we take an uncorrupted dataset and inject it with systematic errors.
These systematic errors are synthetic, designed to seem like reasonable image corruptions, e.g. occlusion or failing of a camera sensor.
The types of systematic errors used across datasets are either lines or squares .
Lines (two diagonal, one vertical, and one horizontal) cross the image from side to side, and may have their color set at random (black / white).
These lines always affect the same pixels, and have thickness of one pixel.
Squares are randomly uniformly placed, so is their ﬁll-in color, with size 6 × 6 pixels.
We use different noise levels so we can study their impact, we use [15% , 25% , 35% , 45%] of dataset.
The systematic error corruption process is done by picking uniformly at random an instance, and then applying that systematic error.
We use a range of trusted set sizes by deﬁning the number of instances labelled per systematic error class (type) and per data class.
The latter is the underlying classes in the dataset, e.g. item labels in
Fashion-MNIST .
For each class, either systematic error or data class, we provide label y for a few instances at random obtaining a trusted set.
We use the range TS size =
[5 , 10 , 25 , 50] labelled samples per class, which results in different trusted set sizes depending on number of classes.
Particularly, we have the trusted set ranges: Synthetic-Shapes with [40 , 80 , 200 , 400] total samples; Frey-Faces with [25 , 50 , 125 , 250] total samples; Fashion-MNIST with [90 , 180 , 450 , 900] total samples.
We create ﬁve examples (different random seeds) per noise level and per trusted set size, and train the models on them.
The results are then averaged.
Table 1 describes datasets and their corruption (error types), number of systematic errors and data classes.
More details on this experimental setup in Annex G.
Comparative Methods Our baselines are VAEs since most of the relevant work in sparse semisupervision is of this type (Ilse et al., 2020; Joy et al., 2020), and the task of repair is related with that of manipulating the reconstruction.
We have four baselines: VAE-L2, CVAE, VAEGMM, CCVAE.
For details on model architecture and hyperparameters see Annex H.
The VAE-L2 model is an unsupervised method tackling the issue of corruption by applying strong regularization ( (cid:96) 2 regularization on weights).
VAE-L2 uses the reconstruction likelihood for detection, more details in Annex
A. CVAE is the supervised version of the semi-supervised M2 model (Kingma et al., 2014), and should have better repair quality than M2.
CVAE uses the reconstruction likelihood as an detection score.
We found a smaller variance for p ( z ) to be beneﬁcial, for details see Annex B. VAEGMM, based on (Willetts et al., 2020), is an improved version of M2 for the sparse semi-supervision setting.
In this setting the M2 model tends to have posterior collapse issues with q ( y | x ) , picking one class over others.
VAEGMM overcomes this issue, improving clustering and classiﬁcation performance.
Hence, we expect competitive detection performance from VAEGMM, for details see Annex C.
The CCVAE (Joy et al., 2020) is a state-of-the-art (SotA) semi-supervised disentanglement model, allowing attribute manipulation in semi-supervised settings.
For repair, we follow the automatic attribute manipulation procedure proposed by (Joy et al., 2020)
(see Annex D).
We adapted their code to our pipeline.
Contrary to Joy et al.
(2020), we found performance was superior when using a large up-sampling coefﬁcient for the classiﬁer, i.e. like β in CLSVAE.
We provide two versions of our model, with and without distance correlation penalty in Section 4.4.
So for CLSVAE-NODC use eq.
(13) as training loss, whilst for CLSVAE use eq.
[Figure] [Figure]
5. D ISCUSSION OF R ESULTS
In Figure 2(a) we show performance as a function of the size of the trusted set, i.e. sweep of trusted set sizes, for a 35% noise level.
Similar performance is seen for other datasets (see Annex J).
Table 2(b) shows the results for all datasets for a 35% noise level and trusted set size of 10 labelled samples per class.
Results for all trusted set sizes and noise levels are found in Annex J, with similar analysis on performance.
Figure 1 shows some examples of image repairs for all datasets (larger version
Annex I).
Additional examples of repairs are seen in Annex K, including inlier instances, 45% noise level, and 5 samples per class (trusted set size) for Synthetic-Shapes since its an easier dataset.
Outlier Detection Looking at Figure 2, we see that on average both CLSVAE (our model) and
CLSVAE-NODC have the highest AVPR, registering the best detection performance, with similar scores.
We see that CCVAE, the previous SotA, has similar detection performance as CLSVAE.
VAEGMM, also semi-supervised, lags behind both likely because its designed for slightly larger trusted sets.
All semi-supervised models (CLSVAE, CLSVAE-NODC, CCVAE, VAEGMM) improve their detection performance as the trusted set grows larger.
CVAE and VAE-L2 do not use a trusted set, and thus have the same performance throughout all the trusted set range in Figure 2(a).
These two observations about the trusted set range are also seen, for all datasets and noise levels, in Annex J. CVAE is supervised, still it shows poor performance, this may be due to: issues linked to (decoder) likelihood-based scores (Eduardo et al., 2020; Lan & Dinh, 2020); poor ﬁtting to the data, thus impacting negatively the score.
VAE-L2 uses a likelihood score, and is unsupervised, so poorer detection performance is understandable.
This highlights semi-supervision as being important in systematic error detection.
VAE-L2 registering good performance in Synthetic-Shapes (see
Figure 2(b)) is likely due to this dataset being easier.
Lastly, we note that CLSVAE tends to have better detection performance in higher noise levels relative to other methods (see Annex J).
Therein,
CCVAE has close to or similar detection performance as CLSVAE.
Automated Repair
In Figure 2, we see that on average CLSVAE (our model) is best at automated repair (lowest SMSE on dirty pixels).
We also see that distortion (repair clean pixels, SMSE) is relatively low, but not the lowest.
This results in CLSVAE overall being the best repair method, not only replacing pixel values of the systematic error, but also inferring correctly the structure of the ground-truth repair (both clean and dirty pixels).
This is conﬁrmed in Figure 1, where reconstructions (repairs) by CLSVAE show the best quality: replacing the error values of affected pixels and recovering the underlying ground-truth, whilst preserving the uncorrupted image portion (i.e. low distortion).
CLSVAE has slightly better repair than CLSVAE-NODC on average, but most importantly, it has better performance stability than CLSVAE-NODC, which can be seen in Annex J for repair (dirty pixels).
Further, in Annex J, repair with CLSVAE is more advantageous relative to other models at higher noise levels.
As expected, semi-supervised models (CLSVAE, CLSVAENODC, CCVAE) improve their repair of dirty pixels as the trusted set increases (see Figure 2(a)).
Both VAE-L2 and CVAE do not use a trusted set, so performance is static.
CCVAE has the ability to perform good repair, registering the second best repair for dirty pixels after CLSVAE, but often with higher distortion.
CCVAE suffers from two issues that account for its worse performance relative to CLSVAE.
For one, looking at Figure 1, CCVAE can sometimes fail to replace the pixel values from systematic errors.
Secondly, and more often, it can fail to recover the underlying ground-truth even when replacing erroneous pixel values.
Similarly, it has difﬁculty preserving the uncorrupted image portion (higher distortion).
So some information about inlier appearance is being lost.
This is explained by the fact that CCVAE latent space is not disentangled regarding the clean and dirty patterns.
CVAE can repair some outliers well, but it fails to deal with other systematic errors, which leads to an overall poor repair performance.
This is maybe due to the binary latent variable used for y making it harder to model multiple systematic errors, for more discussion see Joy et al.
VAE-L2 is able to repair some errors, but overall has worse repair than CLSVAE.
Its strong regularization, optimized mostly for detection, leads to higher distortion and loss of detail (see Figure 1).
Its higher standard error (erratic repairs) is due to not being able to distinguish between clean and dirty patterns.
VAEGMM does not do well in repair.
This is likely due to it being better suited for classiﬁcation or clustering tasks.
C ONCLUSION
We have proposed a novel semi-supervised VAE (CLSVAE) for outlier detection and automated repair, in the presence of systematic errors.
Our model exploits the fact that systematic errors are predictable by high capacity models, unlike random errors.
Thus, CLSVAE partitions the latent space into two subspaces: one for clean patterns, and another for dirty ones.
Inliers are only modelled by the clean pattern subspace, whilst outliers use both subspaces.
We encourage low mutual information between these subspaces through a penalty, improving performance stability.
Empirically this encourages higher ﬁdelity repairs by the model, without human in the loop or other post-processing.
We show CLSVAE only needs a small trusted set, requiring the user to label less data.
We show that unsupervised models may not be able to distinguish between clean patterns and systematic errors, and strong regularization leads to a lower quality repair.
Experimentally, CLSVAE showed superior repair quality and performance compared to other semi-supervised models, including a SotA disentanglement model.
Experiments were carried out on image data, and in the future, we would like to explore other types of systematic errors and data types (tabular, sensor, natural language).
A VAE- L 2 (U NSUPERVISED )
The unsupervised VAE can be used to perform both outlier detection and repair.
However, without regularization it will overﬁt to systematic errors (outliers).
So, here we regularize both the encoder and decoder weights via (cid:96) 2 regularization (weight decay).
We can write the modiﬁed ELBO as
[Equation]
where θ and φ are the parameters for the decoder and encoder respectively.
Further λ (cid:96) 2 deﬁnes the regularization strength.
The w i are the parameters for a layer indexed by i belonging to the set of all VAE layer parameters W AE .
Note that λ (cid:96) 2 is picked by using trusted set performance in outlier detection (AVPR) and repair (SMSE) metrics.
Using repair metrics is not possible in practice, under our problem formulation, since we only have labels about which instances are outliers / inliers.
However we relaxed this assumption here, since we wanted to obtain the best results possible with
VAE-L2.
In terms of outlier detection, we used an anomaly score based on the reconstruction (negative loglikelihood) of the VAE.
This type of anomaly score is typical in VAEs.
For each d ∈ { 0 , ..., D } features, we have
[Equation]
where the outlier set of points is O =
{ x ∈ X|
A ( x ) ≥ γ } , where γ is decided by the user.
The repair process for the outliers is
[Equation]
B CVAE (F ULLY S UPERVISED )
The CVAE (Conditional VAE) model (Kingma et al., 2014; Sohn et al., 2015) needs the entire train set to be labelled, i.e. like an oracle where the ground-truth for y is observed.
This is generally not possible in practice, particularly for our model, as it would be cheating under the problem formulation.
But it does provide a valuable baseline about what kind of performance this type of model, given all advantages, can obtain.
A semi-supervised version of this model is in fact the
M2 model (Kingma et al., 2014), which will always register lower performance than its supervised counter-part the CVAE.
The ELBO for this model is
[Equation]
where θ and φ are the decoder and encoder parameters.
Further, note that in a standard CVAE we usually have p ( z ) =
N ( z | 0 , I ) .
However, we found empirically that stronger regularization for the encoder was needed for CVAE to do well in repair.
Otherwise, it would often overﬁt to the systematic errors.
Hence we used a modiﬁed prior p σ ( z ) =
N ( z | 0 , σ 2 I ) where σ ∈ [0 . 1 , 0 . 8] , thus enforcing stronger regularization (lower message capacity) on latent code z .
Even though it is not standard, we also tried q φ ( z | x ) but always ended up with a bit worse performance than q φ ( z | x , y ) .
In terms of outlier detection, we used an anomaly score based on the reconstruction (negative loglikelihood) of the CVAE model.
This type of anomaly score is typical in VAE type models.
For each d ∈ { 0 , ..., D } features, we have
[Equation]
[Equation]
where the outlier set of points is O =
{ x ∈ X|
A ( x ) ≥ γ } , where γ is decided by the user.
We followed the CVAE attribute manipulation strategy from (Klys et al., 2018) when deﬁning repair of an outlier.
We encode using the original label (i.e. y = 0 ), then switch label in latent space (i.e. y = 1 ) and decode it.
In other words, for repair we have
[Equation]
C VAEGMM: A LTERNATIVE TO M2 (S PARSE S EMI -S UPERVISED )
This formulation is based on Willetts et al.
(2020), speciﬁcally GM-DGM (Gaussian Mixture Deep
Generative Model).
The work focused on severe sparse semi-supervision (some classes do not even have labels).
This formulation is relevant, in sparse semi-supervision, since posterior collapse of q ( y | x ) can occur for the original M2 model (Kingma et al., 2014).
This in turn leads to poor performance in classiﬁcation and clustering tasks.
Hence, one can think of Willetts et al.
as an improvement on M2 model for the issue above.
We now present the version of GM-DGM from
Willetts et al.
(2020) applied to our problem.
C. G ENERATIVE M ODEL
The generative model is deﬁned as
[Equation]
where
[Equation] [Equation]
[Equation]
which deﬁnes a 2-component Gaussian Mixture Model w.r.t
z , and τ = { σ y =1 , σ y =0 } and σ y =1 < σ y =0 .
A sensible range for τ is: σ y =
=
[0 . 2 , 1] ; σ y =
=
This prior deﬁnes a 2-component
Richter distribution (Gales & Olsen, 1999), expressing a type of heavy-tailed distribution on z .
This type of distribution is common in robust statistics (Eduardo et al., 2020), as a means to robustify (regularize) the parameters of a model to outliers.
The main idea here is that inlier samples z | y
=will be regularized more strongly compared to outliers z | y = 0 , reﬂected by σ y =1 < σ y =0 .
We initially tried learning the parameters of the Gaussian components .
However, we obtained much better results by ﬁxing their parameters (i.e. mean vector and covariance matrix), particularly when it came to outlier detection with smaller trusted sets.
Lastly α reﬂects the initial belief on the fraction of clean data.
C. V ARIATIONAL M ODEL
As for the variational model, we use the standard formulation provided in Willetts et al.
(2020), which is the same as the original M2 model (Kingma et al., 2014).
So we have
[Equation] [Equation]
[Equation]
(29) where π φ ( x ) , µ φ ( x , y ) and Σ φ ( x , y ) are neural networks.
C. T RAINING L OSS
We can write the ELBO (Evidence Lower Bound) for the unsupervised part of the dataset, i
[Equation]
e. x ∈ X u , where labels y are not observed as
[Equation]
One can rewrite the above ELBO in a different way as
[Equation]
where
[Equation]
and π φ ( x ) =
q φ ( y = 1 | x ) is the probability a data instance is clean.
The ELBO for the labelled part of the dataset (i
[Equation]
e. trusted set), where we have ( x
, y ) ∈
X l , Y l , is written as
[Equation]
Given the aforementioned deﬁnitions for the model and ELBOs (labelled and unlabelled), the overall training loss is the same as eq.
(13) from the main paper.
C. O UTLIER D ETECTION AND R EPAIR P ROCESS
After training the model, then we can proceed with the outlier detection and repair process.
First, we need to deﬁne a score for use in detection.
So at test time we use
[Equation]
which is based on the probability of the instance being an inlier.
Usually, the user deﬁnes a threshold γ to classify instances into inliers or outliers.
The set of outlier instances is given by
O =
{ x ∈ X| A ( x ) ≥ γ } , where γ ≥ 0 .
Assuming q φ ( y = 1 | x ) is somewhat calibrated, then one can use a γ ≈ − log(0 .
After obtaining the set O , and conditioning on y = 1 , we produce a repair using
[Equation]
D CCVAE (S EMI -S UPERVISED D ISENTANGLEMENT )
In Joy et al.
the latent space z is split into two subspaces: the style (or agnostic) part z \ c that is meant to model unlabelled patterns present in the instance; the characteristics part z c that is meant to model the labelled attributes (characteristics).
Formally, we have z = (cid:2)
z c ; z \ c (cid:3) where
[ ; ] is the concatenation operation.
In our case, following Joy et al.
(2020) and their implementation code, we modelled z c as single variable (univariate) for our binary label:
y = 1 for inliers and y = 0 for outliers.
D. G ENERATIVE M ODEL
From Joy et al.
(2020), using a similar notation, we have
[Equation] [Equation]
[Equation]
where p θ ( x | z ) is a neural network based decoder, and θ are the parameters of the neural network.
In this case α has the same meaning as in CLSVAE, where it expresses the prior belief on the fraction of inliers present in the dataset.
Note that the above model expresses a two-component mixture model on z c , with y as gating variable.
In fact we have a mean and variance for each y value as it pertains to z c , i.e. µ ψ ( y = 1) and σ ψ ( y = 1) for inliers and similar for outliers ( y = 0 ).
D. V ARIATIONAL M ODEL
The variational distribution is factorized as
[Equation]
where q ϕ ( y | z c ) and q φ ( z | x ) are neural network based encoders, with ϕ and φ as neural network parameters.
D. T RAINING L OSS
The training loss used in Joy et al.
(2020) is deﬁned by the ELBO, with one term for the labelled data (trusted set) and a second term the unlabelled data.
The total ELBO is written as
[Equation]
46) where the labelled part of the ELBO is
[Equation]
where β is the hyperparameter controlling amount of up-sampling and importance relative to other terms, like in CLSVAE.
In Joy et al.
(2020), for their application, they found that setting β =brought good results and found no need to tune it further.
In our case, we found that we obtained better performance by using larger values for β .
This is probably because the application is different, i.e. outlier detection and subsequent repair, and since the trusted set (labelled set) is quite small.
The unlabelled part of the ELBO is not as important for our analysis here, please see the original paper.
D. O UTLIER D ETECTION AND R EPAIR
After training the model, we proceed with outlier detection and automated repair.
We need to deﬁne a score for use in detection, and for that we use the classiﬁer given by the variational model.
As such, we have
[Equation]
which is the approximate negative log-probability of an instance being an inlier.
The user may deﬁne a threshold γ to classify instances into inliers or outliers.
The set of outlier instances is given by O = { x ∈ X|
A ( x ) ≥ γ } , where γ ≥ 0 .
Assuming the classiﬁer is somewhat calibrated, then one can use the a γ ≈ − log(0 .
From the perspective of CCVAE, the repair of an outlier instance is just attribute manipulation via z c subspace.
Once the appropriate z c is found, then one uses the decoder for reconstruction obtaining a repair.
Under our problem deﬁnition, automated repair is very much like conditional generation as seen in Joy et al.
(2020), where samples for z c are drawn from the conditional prior whilst reusing z \ c obtained from encoding the outlier instance.
Speciﬁcally, we have z = (cid:2)
z c ∼ p ψ ( z c | y ); z \ c (cid:3) , where depending on the y value it forces the generated samples to have, or not have, the presence of the attribute (e.g. inlier / outlier).
In our case, we are interested in automated repair, and thus limiting human interaction apart from building the trusted set.
That means that exploring z c with user interaction to pick the best repair (reconstruction) for each outlier instance is not realistic.
So we wish to obtain the most likely reconstruction under y = 1 , i.e. the maximum a posteriori , thus deﬁning an automated repair for the outlier instance.
Hence, the repair is given by
[Equation]
where µ ψ ( y = 1) is the mean for the inlier component of p ψ ( z c | y ) , and µ φ ( x ) \ c is the mean of q φ ( z \ c | x ) , which excludes the characteristic (labelled) latent subspace.
E D ISCUSSION ABOUT C LASSIFIER IN CLSVAE
We found that using q φ y ( y | z c , z d ) leads to better performance in more challenging scenarios, e.g. smaller trusted set or when higher data corruption is present.
One could have used q φ y ( y | z d ) , which works, but it is still worse than the former.
This is mostly because z c provides important context on the clean patterns present in the instance, which in turn allows for z d to contain less information about clean patterns.
Hence, z d will be able to focus more on the dirty patterns, and so the mutual information (MI) between z c and z d will be lower.
So the idea of using the stop gradient operator in q φ y ( y | z c , z d ) =
Bernoulli ( y | π φ y ([ sg ( z c ); z d ]) has to do with preventing z c from having dirty pattern information.
By using sg ( z c ) , when executing back-propagation, we prevent the gradient ﬂow from q φ y ( y | z c , z d ) to inﬂuence the learning of parameters φ c .
As such, φ c will only be affect by gradients stemming from the reconstruction of inlier instances, and not the decision on y .
This trick is useful early on in the training, stabilizing the outcome of the optimization procedure, leading to better repair.
Generally, only a few cases beneﬁt from this trick, and usually when small trusted sets are used.
Further, q φ y ( y | z c , z d ) allows for the cross-entropy loss used for the trusted set to directly bias the latent space.
This means labels
y ∈
Y l can help in separating the latent inlier and outlier representations (manifolds).
Other works have tried biasing the latent space in the same fashion, e.g.
(Locatello et al., 2019; Ilse et al., 2020; Joy et al., 2020).
F E MPIRICAL D ISTANCE C ORRELATION
For a random data batch of n samples, such as ( Z , Z (cid:48) )
= { ( z k , z (cid:48) k ) :
k = 1 , ..., n } , one can deﬁne the empirical estimate for the distance correlation between multivariate random variables z ∈ R q and z (cid:
) ∈ R p .
Note that q and p can be of different dimensions, i.e. q (cid:54) = p .
Given this data batch, using the deﬁnition from (Sz ´ ekely et al., 2007), we ﬁrst deﬁne A kl and B kl as
[Equation]
and similarly, using b kl = ||
z (cid:48)
k − z (cid:48)
l || 2 we deﬁne
[Equation]
Now we are ready to deﬁne the empirical distance covariance as
[Equation]
and the following empirical distance variances for each random variable as
[Equation]
Finally, combining the above measures we can write the distance correlation as
[Equation]
where 0 ≤ dCorr n ( Z , Z (cid:48) )
≤ 1 .
Note that if dCorr n ( Z , Z (cid:48) )
=
then z and z (cid:48) are statistically independent random variables.
Otherwise, if dCorr n ( Z , Z (cid:48) )
=
1 then they are strongly correlated.
The distance correlation measure can capture non-linear dependencies, whilst the more common Pearson correlation can only capture linear dependencies.
Further, a value of 0 for Pearson correlation does not imply independence, unlike in distance correlation which it does.
G E XPERIMENTS : D ETAILS ON D ATASETS , C ORRUPTION , AND T RUSTED S ETS
Here we have a detailed description of the datasets and their corruption.
For each noise level corruption in NL size = [15% , 25% , 35% , 45%] , we instantiate ﬁve different examples of the same dataset using different random seeds.
For each of those examples we build several trusted sets using the sizes in TS size = [5 , 10 , 25 , 50] , where labelled instances of smaller trusted sets are reused in bigger ones.
The models are run on each example, for each trusted set, and results are then averaged.
Synthetic-Shapes
This is a synthetic image dataset.
It is meant to test the models in a simpler setting as it relates to the clean dataset.
We treat the pixel values as a Bernoulli variable.
The underlying clean dataset is composed of four different shapes (classes): a circle, a rectangle, an ellipse and a triangle.
The shapes are ﬁlled by white pixels and the background is black.
For each instance, the shape is placed uniformly at random inside the 28 × 28 black background.
The systematic errors have four types, all are white lines that cross the square image from one side to another side.
We have 4 ﬁxed-in-place lines (two diagonal, one vertical, and one horizontal), affecting the same pixels, and sometimes intersecting with the shapes.
Hence, we have a total of eight underlying classes for trusted set constitution, i.e. four data and four systematic error classes.
The size of the images is 28 × 28 , and the pixel values are binary { 0 , 1 } , i.e. black and white.
Overall, we have a dataset size of N = 5000 , with the following split: train ( 80% ), validation ( 10% ), test ( 10% ).
Given the eight underlying classes, the size dataset N , and TS size , we have the trusted set range: [40 , 80 , 200 , 400] total number of labelled instances, which corresponds to [0 .
8% , 1 . 6% , 4% , 8%] of the entire dataset
Frey-Faces This is a gray-scale image dataset consisting of the same person with different facial expressions.
We treat pixel values as continuous.
In terms of data classes, we only have one (monolithic), since no labels for the expressions are provided.
We have four systematic error classes, which consist of four randomly uniformly placed squares of 6 × 6 pixels.
The place and color of these squares is deﬁned by the random seed, when the corruption example is created.
After that, always the same features are affected.
Hence, in total we have ﬁve underlying classes for trusted set constitution, i.e. one data and four systematic error classes.
The size of images is 28 × 20 , and the pixel values range from [0 , 256] , i.e. gray-scale.
The size of the entire dataset is N = 1965 , with the following split: train ( 80% ), validation ( 10% ), test ( 10% ).
Given the above and TS size , the trusted set range is: [25 , 50 , 125 , 250] total number of labelled instances, which corresponds to [1 . 3% , 2 . 5% , 6 . 4% , 12 . 7%] of the entire dataset.
Fashion-MNIST This is a gray-scale image dataset, which consists of images of different types of clothing and accessories from an online merchant.
There are 10 existing data classes, provided with the dataset.
We have 4 ﬁxed-in-place lines (two diagonal, one vertical, and one horizontal) where the color (black or white) for each depends on the random seed.
Then we have 4 squares of size 6 × 6 placed randomly uniformly with random color, dependent on random seed.
Hence, we have a total of 18 underlying classes for trusted set constitution, i.e. 10 data and 8 systematic error classes.
The size of the images is 28 × 28 , and the pixel values are continuous with range [0 , 1] , i.e. grayscale.
The original train set is of size 60000 instances, which we split for our actual train set of( 90% ) and validation set of 6000 ( 10% ).
We use the same test set of 10000 instances, so N = 70000 .
Given TS size and the train set size N , the trusted set range is: [90 , 180 , 450 , 900] total number of labelled instances, which corresponds to [0 .
28%] of the entire dataset.
H E XPERIMENTS : M ODEL A RCHITECTURES , H YPER - PARAMETERS AND O PTIMIZATION Hyperparameter Selection
In our experiments, we tuned model hyperparameters according to outlier detection performance, which means highest AVPR (average precision).
This was evaluated on the trusted set, the only labelled part of the dataset.
Often, we would look at the repairs (reconstructions) offered by each model for the trusted set.
This way we conﬁrm that the repair process is reasonable enough, and no adjustment is needed on the hyperparameters side.
In the case of VAE-L2 we had to not just account for AVPR in the trusted set, but also check repair performance via SMSE (standardized mean squared error).
This is because strong regularization, higher (cid:96) 2 coefﬁcient, often leads to better outlier detection performance, but that comes at the cost of repair quality due to the VAE collapsing to mean behaviour.
For each model, the hyperparameter search was carried out for each dataset at a noise level of 35%, and with 25 samples per class.
• VAE-L
In Synthetic-Shapes : 200 epochs; KL divergence annealing used; (cid:96) 2 coefﬁcient is 35 . 0 .
In Frey-Faces : 300 epochs; KL divergence annealing used; (cid:96) 2 coefﬁcient is 100 . 0 .
In Fashion-MNIST : 100 epochs; KL divergence annealing used; (cid:96) 2 coefﬁcient is 100 . 0 .
• VAEGMM In Synthetic-Shapes : 200 epochs; KL divergence annealing used; fraction of clean data α = 0 .
6 ; (trusted set) up-sampling coefﬁcient β = 1000 ; σ y =1 = 0 .
9 and σ y =
= 5 . 0 .
In Frey-Faces : 300 epochs; KL divergence annealing used; fraction of clean data α = 0 .
6 ; (trusted set) up-sampling coefﬁcient β = 1000 ; σ y =1 = 0 . 6 and σ y
=
= 5 . 0 .
In Fashion-MNIST : 100 epochs; KL divergence annealing used; fraction of clean data α = 0 . 6 ; (trusted set) up-sampling coefﬁcient β = 100 ; σ y =1 = 0 .
5 and σ y =0 = 5 . 0 .
• CVAE In Synthetic-Shapes : 200 epochs; KL divergence annealing used; σ = 0 .
In FreyFaces : 300 epochs; KL divergence annealing used; σ = 0 . 2 .
In Fashion-MNIST :epochs; KL divergence annealing used; σ = 0 .
• CCVAE
In Synthetic-Shapes : 200 epochs
In
-Shapes
correlation
penalty
t
ratio of
• CLSVAE-NODC Same hyperparameter options as CLSVAE but without the distance correlation penalty.
Note the annealing ratio above for the KL divergence and for the DC penalty is inspired by Fu et al.
We use only one cycle (monotonic), and the R (or ratio ) is the proportion used to increase the penalty coefﬁcient (or KL term coefﬁcient) – e.g. 0 .
Optimization
We used the PyTorch framework to code all our models, and trained on a GeForce TITAN X GPU.
All models were trained using the Adam optimizer, with an initial learning rate of 0 . 001 .
Model Architectures
For continuous type data, i.e. Fashion-MNIST and Frey-Faces , we used the Gaussian distribution as the likelihood of each pixel in the reconstruction loss.
The variance of the Gaussian distribution is shared amongst all the pixels in the image, and it is learnt as a parameter of the model.
This was done for all models.
For binary type data, i.e. Synthetic-Shapes , we treated each pixel as a Bernoulli variable, and used the log-likelihood of this distribution for each pixel in the reconstruction loss.
This was done for all models.
We used very similar encoder and decoder architectures for all models, so as to be fair and the results comparable.
In the case of CLSVAE we used two encoders, one for the clean subspace z c , and the other for the dirty subspace z d .
This architecture yielded better results for us in terms of repair in the trusted set.
CLSVAE architecture can be seen in Table 5 for the encoders and decoder, and the classiﬁer can be seen in 7.
In Table 2, we see the neural architecture for the encoder and decoder of
VAE and VAEGMM.
The classiﬁer architecture for the VAEGMM can be seen in Table 6.
In Table
3, we ﬁnd the architecture for the encoder and decoder of CVAE.
In Table 4, we ﬁnd the architecture for the encoder and decoder of CCVAE.
The classiﬁer architecture of CCVAE is just z c multiplied by a weight parameter plus a bias parameter, and then a sigmoid non-linearity is applied – like in
Joy et al.
Encoder
Decoder (img size, 200) → → (15, 50) → → ReLU → → ReLU → → (200, 100) → → (50, 100) → → ReLU → → ReLU → → (100, 50) → → (100, 200)
→ → ReLU → → ReLU → → 2 × (50, 15) → (200, img size)
Encoder
Decoder (img size, 200) → → (16, 50) → → ReLU → → ReLU → → (200, 100) → → (50, 100) → → ReLU → → ReLU → → (100, 50) → → (100, 200)
→ → ReLU → → ReLU → → 2 × (50, 15) → (200, img size)
Encoder
Decoder (img size, 200) → → (16, 50) → → ReLU → → ReLU → → (200, 100) → → (50, 100) → → ReLU → → ReLU → → (100, 50) → → (100, 200)
→ → ReLU → → ReLU → → 2 × (50, 16) → (200, img size)
Encoder z c
Encoder z d
Decoder (img size, 200) → (img size, 200) → → (15, 50) → → ReLU → → ReLU → → ReLU → → (200, 100) → → (200, 100) → → (50, 100) → → ReLU → → ReLU → → ReLU → → (100, 50) → → (100, 50) → → (100, 200)
→ → ReLU → → ReLU → → ReLU → → 2 × (50, 10) → 2 × (50, 5) → (200, img size)
[Table] [Table]
[Figure]
J R ESULTS FOR A LL N OISE L EVELS AND T RUSTED S ET S IZES (A LL S WEEPS )
K A DDITIONAL R ECONSTRUCTIONS (R EPAIRS ) FOR ALL D ATASETS
[Figure]
[Figure] [Figure]
[Figure]
L T ESTING C OMPRESSION H YPOTHESIS : E NTROPY OF C LEAN VS . C ORRUPTED D ATA
In this section we experimentally compare the entropy of clean (without corruption) training data, and the entropy of corrupted training data.
A larger entropy means that a dataset has larger variance overall.
Note that in the setup of our problem, i e. repairing systematic errors (see Section 3), only corrupted data is used for training.
In Figure 22, for Frey-Faces , we compare the estimated entropy of clean training data against one that has been corrupted (35 % noise, corruption as in Table 1)
We estimate the entropy by ﬁrst training a standard VAE model on the dataset (clean or corrupted), and then after training, we compute a tight bound on the marginal log-likelihood of that dataset.
We compute this tight bound via IWAE estimator (importance weighted autoencoders, (Burda et al., 2016)), where we use K = 250 samples.
Note that entropy is H ( x ) = − E p θ ( x ) [log p θ ( x )] , and hence marginal log-likelihood is just −H ( x ) .
We vary the dimension of VAE latent space ( z ) in order to see how well the model can learn the training data.
For this experiment, the VAE is not regularized.
The main idea is to prove empirically that corrupted data has larger variance than clean data, and hence has more information to be modelled.
Speciﬁcally, when using the type of masking corruptions used in our experiments.
If a dataset has larger estimated entropy, then the data can be seen has having larger variance.
A dataset with larger variance is a dataset with more diversity in terms of the patterns it contains, and hence it has more information to be compressed (or modelled).
In particular, in a VAE this means more capacity (e.g. larger latent space) is needed to model a larger variance dataset.
Our claim, supported in literature (Eduardo et al., 2020; Ruff et al., 2019), is that a dataset that has been corrupted has larger variance because the added outliers (e.g. systematic errors) increase data pattern diversity, hence increasing entropy.
Looking at Figure 22, we see that overall the entropy of corrupted data is larger than clean data, for all sizes of z .
Hence, corrupted data has larger variance than clean data.
For the smaller dimensions of z , in range [2 , 10] units, we see that the VAE has less trouble learning the clean data compared to the corrupted data.
This is also evidenced by how more quickly the entropy decreases for clean data relative to corrupt data, as we increase the latent space size in [2 , 10] .
We conclude that a VAE only needs a smaller latent space (subspace) to model clean data (inliers), and that corrupted data needs a larger latent space to be modelled properly.