P RACTICAL I NTEGRATION VIA S EPARABLE B IJECTIVE N ETWORKS
A
BSTRACT
Neural networks have enabled learning over examples that contain thousands of dimensions.
However, most of these models are limited to training and evaluating on a ﬁnite collection of points and do not consider the hypervolume in which the data resides.
Any analysis of the model’s local or global behavior is therefore limited to very expensive or imprecise estimators.
We propose to formulate neural networks as a composition of a bijective (ﬂow) network followed by a learnable, separable network.
This construction allows for learning (or assessing) over full hypervolumes with precise estimators at tractable computational cost via integration over the input space .
We develop the necessary machinery, propose several practical integrals to use during training, and demonstrate their utility.
I NTRODUCTION
Most supervised learning problems operate by training a model on a ﬁnite collection, T , of N (typically paired) examples, ( x, y ) .
The model is updated by comparing its predicted output to the expected output and performing some ﬂavor of stochastic gradient descent based on the comparison and various regularizers.
The process is repeated by reexamining the elements of T in a random order, possibly with augmentation, until the model parameters converge or an iteration budget is exceeded.
This relatively simple procedure has proven to be remarkably effective in a variety of domains and these models have begun to permeate every aspect of modern science and everyday life (
He et al., 2016; Silver et al., 2017; Brown et al., 2020).
The deep learning revolution has also resulted in highly effective generative models such as VAEs (Kingma & Welling, 2014), GANs (Goodfellow et al., 2014), and tractable likelihood models (Dinh et al., 2017; Oliva et al., 2018; Grathwohl et al., 2019).
These models are largely used to create novel samples of impressive quality.
In addition to sampling, likelihood models provide an estimate of the probability density function of the data which can be used for additional, downstream processes.
We augment the training process by constructing neural networks that allow for tractable integration over the input domain .
This differs from implicit layers which utilize integration over a parameterized variable (Chen et al., 2018; Grathwohl et al., 2019).
Access to fast and differentiable integrals allows us to regularize a model’s average behavior using metrics that may not be available otherwise.
Integration over the input space also allows us to supervise how the model behaves in continuous regions that are not directly observed in T and may even be out-of-distribution (OOD).
Alternative methods attempt to supervise examples outside of T by performing random perturbations (Gutmann & Hyvärinen, 2010), along the line between known examples (Zhang et al., 2018), or via a generative process (Zenati et al., 2018; Akcay et al., 2018).
However, these methods are only capable of observing a small quantity of the total space.
By integrating over entire regions, it is possible to observe a large portion of the space based on statistical relevance.
Main Contributions
We propose a general architecture that enables tractable integration over the input space, enabling supervision and custom regularization over continuous regions.
We demonstrate how to construct this network and how it allows for a reduction in the computation cost required for dense numeric integration from exponential in the number of dimensions to linear.
We derive several useful integrated formulations over continuous regions.
Finally, we explore the impact of this architecture and regularizers on the standard accuracy and robustness to OOD examples on several standard classiﬁcation datasets.
Notation Throughout this work, we consider the M dimensional input features, x = [ x 1 , ..., x M ] ∈
R M ; the latent features, z ∈ Z ⊆ R M ; and the K -wise classiﬁcation probability, y .
The input features x the training set, T x , are drawn from the in-distribution data, D ⊆ R M .
Subscripts represent a particular dimension, e.g., D m corresponds to the m th dimension of the space.
Paranthetical superscripts represent the subspace corresponding to a particular class, e.g., D ( c ) is the subset of
D where the data belongs to class c .
The bijective network is given as h such that h : D → Z .
Probability distributions over D and Z are given by p with the corresponding subscript.
Classiﬁcation networks are given as f and g .
Variables with a “hat,” ˆ y , are predictions of the true quantity, y .
M OTIVATION
Neural networks are highly effective function approximators between two (typically) continuous spaces: f : X → Y .
However, networks are typically trained and evaluated using a ﬁnite collection of points without any explicit assessment of the complete hypervolume spanned by the data.
This omission is understandable from an implementation perspective as the number of samples required to obtain a reasonable estimate over a volume scales exponentially with data dimensionality.
However, human beings often have an understanding of how a process should behave on average .
Ideally, we would like to embed this intuition into the model but currently cannot assess the average performance of a trained model outside of the held-out test set.
Speciﬁcally, we would like to regularize the model by estimating the expected behavior of some metric, Ω , produced by the model over the training data
[Equation]
There are many useful choices of Ω over a variety of applications.
If it is known what the model output should be on average ( ¯ y ), we can construct Ω to encourage that behavior, e.g., Ω(ˆ y )
= (¯ y − ˆ y ) 2 .
Minimizing consistency metrics (Xie et al., 2020) are a common method to improve learning in label-starved problems.
These encourage the model to produce similar outputs over neighborhoods around (labeled) examples from T x where neighborhoods are created by random or domain-speciﬁc augmentations.
This process can be viewed as an approximation to an integral over the neighborhood,
[Equation]
where L is a distance-like metric, and (cid:15) is the neighborhood.
Equation 2 can be generalized to other neighborhoods.
We can recast the standard classiﬁcation problem as a discrete approximation to an integral.
Typically, we minimize the cross-entropy between ˆ y ( x ; θ ) and y over the model parameters, θ , for all ( x , y ) ∈ T which becomes an integral over class-conditioned distributions, p ( x | c ) , [Equation]
[Equation]
Unfortunately, integration in high dimension is difﬁcult.
Naive gridded solutions require an exponential number of points with error decreasing as O ( G − M ) , for G , M -dimensional points.
Monte Carlo (MC) methods theoretically have better performance with error that decreases as O ( G − 1 / 2 ) .
However, the rate of convergence for MC methods depends on the variance of samples (Veach, 1998), which may make for poor approximations in practice.
Importance sampling (Bishop, 2006) can improve the performance of Monte Carlo methods to adapt to the regions with the greatest contribution.
We choose to model the data using a separable function.
Separable functions have the key beneﬁt of decomposing M -dimensional integrals into a combination of M one-dimensional integrals.
Each of these one-dimensional integrals can then be solved using any number of highly-accurate solvers
The use of a separable function over the input space may make estimation of integrals over the model more accessible; however, they impose a strong, inappropriate inductive-bias.
The obvious approach of utilizing a standard neural network as a feature extractor and integrating over learned features means that we would no longer have a valid integrator over the input space.
We propose to solve this problem by utilizing bijective transforms prior to the separable network. The bijective transform allows us to decouple the data into a latent space where the data can be modeled using a separable network and guarantees equality between integrals in the latent space and integrals in the input space.
B ACKGROUND
We perform integration over the input space by splitting neural network models down into two key components: (1) a bijective feature extractor, (2) a separable task network, see Fig.
For simplicity, we only consider classiﬁcation tasks in this work.
This makes our total network analogous with the common architecture where a classiﬁer, often a linear layer or an MLP, is constructed on a feature extractor, such as a CNN.
Unlike the typical process, we must place constraints on both networks so that we can integrate over the input domain.
This network breakdown is similar to hybrid networks (Chen et al., 2019; Nalisnick et al., 2019b) except for the separability requirement on the classiﬁer.
3. B IJECTIVE N ETWORKS
Bijective networks are the key component in ﬂow-based likelihood models.
A bijective network, h : D → Z , has a known forward and inverse operation so that data can exactly be reconstructed after the transformation.
This allows for exact likelihood estimation via the change of variables formula:
[Equation]
where p Z is a predeﬁned distribution over the latent space, often a standard Gaussian.
These models are trained via Eq.
4 to maximize the likelihood of the examples in T .
Once trained, ﬂow-based likelihood models are commonly used as a generative process where samples are drawn from p Z and are then inverted through h to arrive at an example in D .
Instead, we will take advantage of the fact that we can choose p Z and then utilize it in downstream tasks.
Normalizing ﬂow bijectors provide rich feature extractors that can represent a distribution of complicated inputs with simplydistributed, independent features, z .
Given the independence and expressibility of these learned independent features, we build estimators using separable functions over z , which enables us to integrate over the data’s domain while retaining expressibility.
The requirement for bijectivity places a strong constraint on network design, eliminating many common choices due to the need to maintain dimension or invert element-wise activations.
Even naive convolutional operations become unavailable since they are not generally invertible.
Modern advances have demonstrated methods to work around these limitations through the use of clever partitioning and coupling tricks Dinh et al. (2017) or the use of constraints Chen et al.
However, the ﬁeld of learnable bijective functions is less advanced then its unconstrained counterparts which results in reduced performance on auxiliary tasks.
We utilize Glow Kingma & Dhariwal (2018) to process image data and continuous normalizing ﬂows Grathwohl et al. for tabular data.
3. S EPARABLE F UNCTIONS
Separable functions have long been used in mathematics and physics to solve simple partial differential equations such as the homogeneous wave and diffusion equations (Strauss, 2007).
We consider two types of separable functions, additive and multiplicative.
All proofs can be found in Appendix A.
3.2. A DDITIVELY S EPARABLE F UNCTIONS
Deﬁnition 3.1.
A function, f : C M → C K , is additively separable if it is composed as a summation of element-wise functions operating independently on each dimension of the input:
[Equation]
Theorem 3.1 (Additive Independent Integration) .
Given an additively separable function, f ( v ) , an independent likelihood, p ( v ) =
(cid:81)
Mm =1 p m ( v m ) , and domain, D v = D v 1 × ...
× D
[Equation]
M :
[Equation]
3.2. M ULTIPLICATIVELY S EPARABLE F UNCTIONS
Deﬁnition 3.2.
A function, g : C M → C K , is multiplicatively separable if it is composed as a product of element-wise functions operating independently on each dimension of the input: g ( v ) =
M (cid:89) m =
g m ( v m ; ψ m ) (7)
Theorem 3.2 (Multiplicative Independent Integration) .
Given a multiplicitively separable function, g ( v ) , an independent likelihood, p ( v ) = (cid:81) Mm =1 p m ( v m ) , and domain, D v = D v 1 × ...
× D v M :
E v ∼ p ( v ) [ g ( v )] =
M (cid:89) m = (cid:90)
D vm g m ( v m
[Equation]
M ETHOD
Both forms of separable functions allow us to decompose a single M -dimensional integral into M 1 -dimensional integrals.
A dense estimation of the integral without taking advantage of a separable function using G points per dimension would require O ( G M ) network evaluations.
This is completely impractical for modern datasets where M is at least on the order of hundreds and G should be as large as possible.
Exploiting separable functions allow us to reduce the complexity to O ( GM ) .
However, it is unlikely that we could construct a separable function directly on the input space and achieve reasonable performance.
Doing so would essentially require that each input dimension contribute to the ﬁnal output independently of the others.
Instead, we can combine Eq. 4 and Eq.(alternatively, Eq.
8) to perform practical integration over the input domain while still allowing for inter-dependent contributions from each input dimension.
To do so, we ﬁrst let the latent distribution, p Z , be independently (but not necessarily identically) distributed: p Z ( z ) =
(cid:81)
m p m ( z m ) .
This allows us to write the integral over the input space in terms of the latent space and then simplify via Eq. 6.
[Equation]
Each 1-dimensional integral can be easily approximated using a variety of integration approaches.
In addition to the requirements placed on the feature extractor and task network, this formulation also requires that we deﬁne the integration domain in the latent space as a Cartesian product of domains over each latent dimension.
This may seem like a strong requirement since we apparently lose some level of interpretability; however, there are several advantages to deﬁning the input domain in this learned space.
Most notably, we know the data distribution in this space exactly and can tune the domain based on the goal of a particular integral.
[Figure]
Figure 2 contains a cartoon example that demonstrates how these methods combine to allow for integration over the complex data space.
Both plots show the value of f ; Fig.
2a shows the nonseparable function in the input space and Fig.
2b shows the same data in the (separable) latent space, after the bijective transformation.
The colored points and dotted lines are in correspondence between the two spaces and illustrate how the space is warped by the bijector to create a separable, independent latent space.
The light gray lines represent the contours of the underlying probability distribution.
We deﬁne both the integration domain and perform the integration in the latent space.
For simplicity, we illustrate the integration using ﬁve, equally-spaced points (in both dimensions).
The naive procedure would require sampling f at each point in the grid (the blue points).
The use of the separable functions allow us to integrate using only the red points and get the same estimate.
P RACTICAL A PPLICATIONS
In this section we discuss speciﬁc choices made when constructing the latent distribution, separable networks, and various integrals.
For classiﬁcation tasks, it is not possible to parameterize the normalized network output of class probabilities, ˆ y , as a separable network since each prediction must sum to one.
However, it is possible to parameterize each unnormalized logits as a separable network, e.g., ˆ y ( x ) =
σ ( f ( h ( x ))) , where σ is the softmax operator and the output of f are the unnormalized logits.
While this limits what quantities can be integrated over exactly, it is still possible to integrate over approximations/bounds without resorting to brute-force methods.
Out-of-Distribution Detection We construct a global integration regularizer to provide resilience to out-of-distribution (OOD) examples.
We enable OOD detection by introducing a “reject” option (Hendrickx et al., 2021) into the classiﬁcation vector, increasing the number of classes by one, where the additional class indicates that the instance is OOD.
An example is considered OOD if ˆ y K +1 exceeds a predeﬁned threshold.
We supervise the model over out-of-distribution examples by integrating the cross-entropy over the contrastive probability distribution, q ( z ) (see Sec. 5.3.1).
Semi-supervised Learning We build a local integral to enhance consistency within latent neighborhoods and utilize it in place of pre-chosen augmentation strategies to inform a label-starved dataset.
Speciﬁcally, we introduce a loss where all examples near a real, labeled example are regularized to share the real example’s label as in Eq.
2 (see Sec. 5.3.2).
We additionally use pseudo-labels Lee (2013) so that consistency is maintained about unlabelled points as the model grows more conﬁdent.
5. L ATENT D ISTRIBUTIONS
A critical component of this method is that the likelihoods over the latent space must be independent:
p Z ( z ) =
(cid:81) m
p Z ( z m ) .
As this is extremely unlikely to occur in the input space, we learn a bijective (ﬂow) transformation from the input space to a latent space where the likelihood can be decomposed into independent components of our choice.
Since we would like to use the latent features to discriminate between classes using a separable function, we choose to utilize a (bimodal) Gaussian mixture model.
This allows the model to put different classes into one of two “buckets” per feature and enables classiﬁcation through multiple features.
This results in an exponential number of components (with respect to dimension) in the full latent space, e.g., 2 M components.
However, we can easily offset this by choosing some dimensions to use a unimodal latent distribution while the remaining dimensions are still bimodal:
[Equation]
In addition to the typical latent data distribution, p Z ( z ) , we explicitly include a contrastive distribution, q ( z ) .
This distribution is critical if we desire to regularize our model outside the data distribution, i.e., setting some background/default behavior.
When using a bimodal data distribution, we additionally desire that latent vectors are more likely under this distribution once the vector is sufﬁciently far away from any single in-distribution mode.
Therefore, we select this distribution so that it fully encapsulates all the components of the data distribution, e.g., q m ( z m ) > p Z m ( z m ) for | | z m |− µ | > γ .
When the data distribution is unimodal we choose p m =
q m so that the feature is uninformative.
We utilize a standard Gaussian for the contrastive distribution and set µ to 1 and experiment with σ 1 ≤ 0 .
This choice allows us to formulate OOD examples that exist between the in-distribution data as near zero, and outside the in-distribution data, near the tails.
Figure 3 illustrates the data and contrastive distributions for a latent feature for convenience.
5. S EPARABLE N ETWORKS
We explore three different formulations of separable networks.
The ﬁrst formulation learns a distinct quadratic function, the second learns a symmetric hinge, and the third learns a multi-layer perceptron.
These functions have distinct parameterizations for each latent feature and each in-distribution class.
The out-of-distribution logit is held ﬁxed at zero:
[Equation]
where k is the logit class index and m is the feature dimension so that the total unnormalized logit, l k ( z ) , is l k ( z )
= (cid:80) m f k,m ( z m ) .
The separable MLP is constructed by concatenating a learned vector per feature and class to each 1-dimensional feature and constructing a standard MLP that operates on that augmented state.
This formulation provides the greatest ﬂexibility as it leverages all the beneﬁts of the universal approximator theorem (Hornik, 1991).
Unfortunately, we ﬁnd the MLP version difﬁcult to train in practice, often resulting in degenerate solutions.
Fixing the OOD logit at zero provides a ﬁxed reference point for the other logits.
We interpret this as each latent feature voting on how in-distribution the example is on a per-class basis.
5. I NTEGRALS
The cross-entropy, CE (ˆ y ( x ) , y ) , between the normalized model output, ˆ y ( x ) , and the true labels, y , is commonly used to train classiﬁcation models.
In this section, we explore global and local integrals of the cross-entropy loss used to train most classiﬁers (see Eq. 3) when ˆ y is given as the composition of the softmax function, a separable function and a bijective function, e.g., σ ( f ( h ( x ))) .
We can express this as min θ (cid:80) x ,y ∈T CE (ˆ y ( x ) , y )
owing to the ﬁnite number of elements in T .
Ideally, we would minimize the model cross-entropy over all possible examples in D .
We accomplish this by deﬁning the subspace, D ( c ) ⊂ D , that has a given label and corresponding conditional distribution, p ( x | c )
[Equation]
It is not possible to represent ˆ y ( x ) as separable network due to the softmax operation but it is possible to parameterize the unnormalized logits as a separable network.
Substituting this parameterization into Eq. 13 and transforming to the latent space yields a bound for the expected cross-entropy
[Equation]
See Appendix E for a full derivation.
We will utilize this formulation in conjunction with a contrastive prior to supervise OOD examples, Sec. 5.3.1, and with a local prior to enforce consistency, Sec. 5.3.2.
5.3. O UT - OF -D ISTRIBUTION S UPERVISION
We can utilize the cross-entropy integral in different ways by choosing the label we would like to apply over a domain.
For example, we can integrate the cross-entropy over the OOD latent space,
U , with the true label ﬁxed to the reject class ( c = K +1 ), and the latent distribution over codes is the contrastive distribution (Sec. 5.1), p ( z | c = K +1) =
q ( z ) ; using Eq. 14 with f K +1 ,n =0 :
[Equation]
In effect, Eq. 15 discourages OOD data from being labeled as any in-distribution label.
Since the data and contrastive distributions overlap, the model could degenerate and always make OOD decisions; however, this would be in conﬂict with the standard cross-entropy loss applied to each example in the training set.
So long as L GLBL is not weighted too strongly, the model achieves equilibrium by making OOD predictions over regions where the contrastive distribution is more likely.
In effect, we supervise OOD training without requiring any OOD data.
5.3. L OCAL C ONSISTENCY
We may also perform integration over neighborhoods of data points in T x and utilize that point’s label over the entire region.
This integral serves to improve consistency around observations.
Adversarial attacks (Goodfellow et al., 2015; Madry et al., 2018) are often constructed as perturbations on real data points and adversarial training ﬁnds one such point and includes it in the training set.
We can interpret local consistency integrations as a form of average adversarial training.
We reformulate the integral to be centered around a particular datum, x 0 and its class label, y
[Equation]
A complication resulting from local integration is how to select the local distribution, p V ( v ) , and the neighborhood, V .
There are many reasonable choices depending on the goal of the local integration.
For simplicity, we choose each V m ∈ [ − ε, ε ] and p m ( v m ) to be uniformly distributed.
5. L OSS C OMPONENTS
The ﬁnal training loss used to evaluate the model is composed of different combinations of the standard cross-entropy loss over class predictions, L CE = − (cid:80)
k y k log (ˆ y ) , the negative log-likelihood loss over in-distribution data points, L NLL = − log( p Z ( h ( x )))
− log (cid:12) (cid:12) ∂h∂ x (cid:12) (cid:12) , and the integration losses,
L GLBL and L LCL .
We always include L CE and L NLL .
Based on results from previous hybrid networks (Chen et al., 2019; Nalisnick et al., 2019b), we weight the negative log-likelihood by 1 /M , which is analogous to using bits per dimension and introduce an additional weight over the cross-entropy, λ , which controls the relative importance of the generative and classiﬁcation tasks.
When included, each integration penalty shares the same weight as L CE with additional binary weight, π
[Equation]
R ELATED W ORK
Noise Contrastive Distributions Noise contrastive methods introduce a distribution that is distinct from the data distribution, preferably with minimal overlap.
The constrastive distribution can be learned and provides a mechanism to supervise the model to discriminate between true and contrastive samples (Gutmann & Hyvärinen, 2010).
This forces the model to learn discriminative statistical properties of the data.
We utilize this idea to inform the null-space over which we will integrate.
We ﬁx the data and contrastive distributions and learn the bijective map between input and latent spaces.
Hybrid Models Normalizing ﬂows are a family of ﬂexible, tractable likelihood estimators based on the application of learnable, bijective functions Dinh et al. ; Grathwohl et al.
These methods have shown strong performance, both as likelihood estimators and as generative processes.
Recent work has coupled advances in bijective networks to construct invertible feature extractors that are capped by a more conventional classiﬁer.
The combination bijective/classiﬁer structure are called hybrid models and show reasonable performance as likelihood and discriminative models Chen et al. ; Nalisnick et al. (2019b).
Unfortunately, the discriminative performance of these models is lower than traditional methods.
We utilize hybrid models with constraints that enable tractable integration.
Other hybrid models architectures are less restricted but prohibits practical integration.
Out of Distribution Detection OOD detection is a challenge for many modern ML algorithms.
Recent work has demonstrated that OOD data can achieve higher likelihoods than the training data Hendrycks et al. ; Nalisnick et al. (2019a).
Several recent methods have been developed to detect OOD examples including Maximum Softmax Probability (MSP) Hendrycks & Gimpel (2017), Outlier Exposure (OE) Hendrycks et al.
Multiscale Score Matching (MSMA)
Mahmood et al. (2021), ODIN Liang et al. (2018), and Certiﬁed Certain Uncertainty (CCU) Meinke & Hein (2020).
Lee et al.
(2018) utilize a GAN trained with a classiﬁer to produce examples near but outside the training distribution.
These methods are constructed speciﬁcally for OOD detection whereas our method is applicable to a variety of problems.
Semi-supervised Learning Semi-supervised learning is a burgeoning research area that learns from a large data corpus when only a small subset are labeled. The problem setting is very pertinent as it is often easy to acquire data examples but can be extremely time consuming to create the corresponding labels. Many modern methods can achieve very strong performance with very few labels Zhang et al. ; Chen et al.
However, most of these methods rely on domainspeciﬁc augmentation strategies that are difﬁcult to replicate in new data regimes, i.e., for non-image data.
Fortunately, methods such as SSVAE Kingma et al.
and VIME Yoon et al.
(2020) are domain agnostic.
These methods are built exclusively for semi-supervised learning but we only require an additional regularizing penalty to achieve comparable performance on tabular datasets.
E XPERIMENTS
[Figure]
All models were constructed using PyTorch (Paszke et al., 2019), trained using PyTorchLightning (Falcon, 2019), utilized bijectors and distributions from Pyro (Bingham et al., 2018), and were trained using Adam (Kingma & Ba, 2015).
We assess the integrable model’s performance in a semi-supervised regime and against OOD examples.
See Appendix B and C for additional experiments and training details.
7. S PIRALS
We construct a synthetic, 2D dataset composed of three intertwined spirals, see Fig. 4a.
Suppose that, due to an unknown sampling bias, we only observe two of the three arms (missing the green arm) and constructed our model as a two-class problem with a third, reject class.
We sample points in a dense 2-dimensional grid at twice the range of the data in both dimensions and evaluate the probability that each point belongs to the two known classes and the OOD class.
Figures 4d and 4e illustrate the probability of belonging to the two known classes and Fig.
4f contains the probability that each point is OOD.
Red and white values indicate high and low probabilities, respectively.
The model successfully identiﬁes the two in-distribution regions.
Unsurprisingly, the model also identiﬁes data outside of the training data range as being OOD and, impressively, identiﬁes the unobserved spiral and the region between spirals as being OOD.
Finally, we sample a square box (orange) around a data point (blue) in the latent space and invert the box to assess the appropriateness of the neighborhood in the input space and plot the result, Figs. 4b and 4c.
As expected, the bijector converts the latent Euclidean neighborhood into a semantically meaningful neighborhood.
Had we included the local cross-entropy integral in this training process, all points within the orange boundary would have received the same label as the data point.
7. O UT OF D ISTRIBUTION D ETECTION
We test how our models respond to OOD examples when trained with global integrals over the noisecontrastive prior and consistency integrals and compare to methods designed cfor OOD detection.
See Appendix B.3 for standard performance and Sec. 6 for a discussion of the baselines.
Table 1 contains the AUPR for the various methods against similar but different datasets (see Appendix
B.4 for the AUROC).
We juxtapose SVHN vs CIFAR10 and MNIST vs FMNIST plus Extended MNIST (EMNIST)
(Cohen et al., 2017).
We include a separable, hybrid model without integral regularizations (hybrid) and the same model with regularizations (Int. Reg.) to illustrate the utility of learning over hypervolumes.
When MNIST or FMNIST are the in-distribution set, the regularized, integrable network performs on par with the best baseline methods.
SVHN and CIFAR10 show reasonable OOD performance but are not as strong as the baselines.
This is not surprising since the integrals rely on a reasonable estimate of p ( x ) , which both datasets have failed to achieve, Table 5. [Table] [Table]
We apply the local integral to the separable hybrid model and train on several tabular datasets with 10% of the labels and domain-speciﬁc augmentation strategies are unavailable.
These models do not utilize the OOD class or global penalty. We apply pseudo-labelling with thresholds of 0.9-0.95. Table 2 compares the performance of the integrated hybrid models to several standard (non-image) semi-supervised baselines on ﬂat MNIST (MNIST, ﬂattened to a vector), MiniBooNE Bazarko (2001) [Table]
and HepMass Baldi et al.
We utilize a CNF Grathwohl et al.
as the bijector and the hinge as the classiﬁer.
We see that the integrated model achieves similar-to-better performance than the other methods on all datasets, showcasing the generality of integrated regularizers in different problem spaces
[Table]
C ONCLUSIONS
In this work [Table] we develop architectures that enable tractable integration over the data space.
We demonstrate that the ability to supervise regions and not isolated points encourages the model to learn better representations and be less likely to degenerate.
We consider several formulations that allow us to regularize the model’s behavior based on consistency and contrast without relying on augmentations of and sparse comparisons between a ﬁnite collection of data points.
We experiment with the various integrals to obtain promising out-of-distribution detection.
Through now tractable integrals, this work enables future methods and applications for learning over continuous regions.
A. A DDITIVELY S EPARABLE F UNCTIONS
Theorem A.1 (Additive Independent Integration) .
Given an additively separable function, f ( v ) , an independent likelihood, p ( v ) =
(cid:81)
Mm =1 p m ( v m ) , and domain, D v = D v 1 × ...
× D v M
[Equation]
Proof.
[Equation]
Substituting Eq.
A1 into Eq.
A3 reduces the M -dimensional integral over independent likelihoods into a 1-dimensional integral and completes the proof.
A. M ULTIPLICATIVELY S EPARABLE F UNCTIONS
Theorem A.2 (Multiplicative Independent Integration) .
Given a multiplicitively separable function, g ( v ) , an independent likelihood, p ( v ) = (cid:81) Mm =1 p m ( v m ) , and domain, D v = D v 1 × ...
× D v M :
E v ∼ p ( v ) [ g ( v )] =
M (cid:89) m
[Equation]
Proof.
Since each component of g n is matched to a component in p m , the two products can be recombined to isolate like-dimensions.
Each dimension can then be integrated independently.
[Equation]
B A DDITIONAL E XPERIMENTS
All image datasets were trained using Glow as the bijector with the number of levels adjusted for the size of the image.
MNIST and Fashion MNIST were trained with Adam for 50 epochs with an exponentially decaying learning rate of γ = 0 . 75 .
SVHN and CIFAR10 utilized a similar training procedure but with 75 epochs and a slower exponential decay of γ = 0 .
We ﬁnd that utilizing an average instead of a sum over separable features and soft-thresholding at − 1 improves training stability.
B. A DVERSARIAL R OBUSTNESS
We test the efﬁcacy of the local cross-entropy integral by utilizing a synthetic dataset from (Tsipras et al., 2019).
The data is constructed via
[Equation]
and the adversarial examples are constructed analytically by, essentially, ﬂipping the sign of η .
We utilize this synthetic dataset to test adversarial robustness because it provides an upper bound on the adversarial accuracy relative to the standard accuracy which allows us to quantify our performance with respect to an absolute.
Additionally, because adversarial examples can be constructed exactly without relying on an optimization procedures, there can be no concerns that this defense relies on obfuscated gradients (Athalye et al., 2018).
We refer the interested reader to the original paper for a thorough discussion.
[Table]
We choose M = 10 , p = 0 .
95 , and η = 2 / √ M .
For these choices, a strong classiﬁer will obtain near perfect standard accuracy with zero adversarial accuracy while a robust classiﬁer will obtain 95% standard and adversarial accuracy.
We use a similar architecture to that found in Sec.
C.1 except we utilize ten blocks instead of ﬁve.
We train one model with only the standard crossentropy and negative log-likelihood losses and a second model with the additional global and local cross-entropy integration loss.
We ﬁnd that the model with only the standard losses achieves reasonably good standard performance and fairly poor adversarial accuracy.
The model with the additional integration losses contains considerably better adversarial performance, nearing the upper and, necessarily lower, standard accuracy bound.
Table 3 summarizes our results and the ideal performances.
B. T OY S EMI - SUPERVISED R EGRESSION
[Table]
To illustrate the utility of global regularizations, we construct a one-dimensional, semi-supervised problem with x ∼ N (0 , 4) and y = tanh( x ) .
We keep all values of x but remove y values corresponding to negative x during training.
Unlike the standard semisupervised problem, the missing labels are not random but are the result of limitations or bias in the data collection process.
We train two models that are identical except that the integrated model includes a penalty over E [Ω (ˆ y ( x ))] based on foreknowledge that the average value is zero.
Tableshows how the models perform over the supervised and unsupervised regions.
The inclusion of the integration regularizer allows the model to reason about regions that it has not directly observed and has decreased the error in that region by two orders of magnitude.
B. S TANDARD P ERFORMANCE
[Table]
We test the impact of integrable models on OOD performance against several standard image classiﬁcation datasets: MNIST (Deng, 2012), Fashion MNIST (FMNIST) (Xiao et al., 2017), SVHN (Netzer et al., 2011), and CIFAR10 (Krizhevsky et al., 2009).
See Appendix B for architures, distributions, and training parameters.
Table 5 contains the validation standard accuracy and bits per dimension for all datasets with all integration regularizers.
The upper portion of the table is averaged over 10 runs and contains a reject option in the separable model.
The lower portion is a single run without a reject option or any integrated regularizers.
We ﬁnd that the model performs reasonably well for MNIST and
FMNIST but the integrated losses cause a large degredation in performance for SVHN and CIFAR10.
The removal of these losses produces similar accuracies but much-improved BPD, consistent with the hybrid network results reported by ResidualFlows (Chen et al., 2019) when Glow is used.
B. O UT OF D ISTRIBUTION D ETECTION AUROC C OMPARISONS
[Table]
I NTERPRETABILITY
The separably model and independent latent dimensions allows us to reason about how the model is making decisions in the latent space.
Unfortunately, for most bijectors, it is not possible to carry this interpretability back to the input space.
Figure 5 demonstrates the ﬁnal state of the model trained on Fashion MNIST for several features with respect to the latent distribution, per in-distribution class and juxtaposed with the out of distribution data.
Speciﬁcally, the top row contains the logit components, f k,m , learned by the separable network, color-coded by class; the middle row contains the distribution of each class (colors matched to logits); and the bottom row contains the distribution of the in-distribution data (blue) and OOD data (red).
The top row illustrates how the classiﬁcation network makes its decisions per feature over the distribution presented in the middle row.
We see that the logit components map well to the distribution of the data in each feature and provides some intuition for how certain the classiﬁer is over a feature.
This demonstrates how this architecture allows for reasonable interpretibility from the latent space.
Any value above zero (the dotted black line) is considered in-distribution and the most likely class is the line with the greatest value at that point.
The features were chosen to demonstrate the diversity of the learned solution over features.
Generally, the data maps reasonably well to the bimodal distribution, though we do occassionally see mode collapse as in Fig.
5e.
Fortunately, in these cases the logits tend to be fairly uninformative and only introduce a small bias.
Figures 5a through 5c show a common trend where the OOD data has heavy overlap with one of the two clusters but not the other.
While we do see some diversity amongst the in-distribution classes that overlap with the OOD data the “bags” (gray) and “sandals” (cyan) class overlap most often.
Finally, Fig.
5d demonstrates a latent feature where the OOD data falls in the region between the two data components. [Figure]
B. MNIST L EAVE - ONE - OUT
Following (Ahmed & Courville, 2020), we test the robustness of our method against semantic OOD examples by training models on all but one class from MNIST (Deng, 2012) and assessing OOD performance on the held out class.
We repeat the process for each class and report the AUROC and
AUPR.
In all cases, we train using ten different random seeds and report the average performance and standard deviation across seeds that do not degenerate against the in-distribution validation set.
Table 8 and Table 7 contain the results of our method using both integrated losses compared to several relevant baselines.
Results from Efﬁcient GAN-Based Anomaly Detection (EGBAD)
(Zenati et al., 2018) and GANomaly (Akcay et al., 2018) are taken by estimating the results from ﬁgures in both works.
The Semantic Anomalies (Ahmed & Courville, 2020) results are obtained by executing the ofﬁcial version of the code using the rotation auxiliary task with two different batch sizes (128 and 256).
The Semantic Anomalies results in both tables are the best across both batch sizes and detection methods (ODIN (Liang et al., 2018) and MSP (Hendrycks & Gimpel, 2017))
based on the AUPR.
We see that the Semantic Anomalies generally achieves the best AUROC across all digits.
The integration often achieves the best AUPR but struggles with certain held out digits.
In particular, the integration performs signiﬁcantly worse on the “4” and “9” digits.
This is a reasonable confuser due to the similarities and variability of the two classes.
These results indicate that the integration penalties are helpful for inducing the model to detect semantic anomalies.
[Table]
C T RAINING AND A RCHITECTURES C. S PIRALS
The bijective layer is composed of ﬁve blocks where each block contains an afﬁne-coupling layer (Dinh et al., 2017) and an invertible fully connected layer.
The data distribution is bimodal over both latent dimensions with means at ± 1 and standard deviations of 0 . 4 .
The separable network is composed of quadratic functions.
We train the model as discussed, using the standard cross-entropy [Table] loss for each observed point, the negative log-likelihood over the input space, and the global crossentropy integration with respect to the contrastive prior.
The learning rate is set to 0.001 with standard weight decay of 1e-4 over 20 epochs using Adam (Kingma & Ba, 2015).
C. F ASHION MNIST
The bijective layers are composed of the Glow (Kingma & Dhariwal, 2018) architecture with two levels composed of 32 steps and 512 channels.
We utilize an open-source Glow implementationwrapped in Pyro’s (Bingham et al., 2018) bijective operators.
The data distribution is bimodal over all latent dimensions with means at ± 1 and standard deviations of 0 . 5 .
The noise constrastive distribution is a standard Gaussian.
The separable network is composed of hinge functions (see Sec. 4.3).
We utilize a batch size of 256, a learning rate of 1.5e-4 over the bijective layers, and a learning rate of 1.0e-3 over the separable layers using Adam (Kingma & Ba, 2015).
Both learning rates are exponentially decayed at a rate of 0.75 per epoch.
Standard weight decay is applied with a weight of 1e-4.
The network is trained for 50 epochs.
The adversarial attacks are performed against the model at the epoch with the lowest validation cross-entropy loss.
D A PPROXIMATE E XPECTED C ROSS -E NTROPY OVER A D OMAIN
We desire the expected cross-entropy, CE (ˆ y ( x ) , y c ) , over a domain D c where y c is the desired probability vector over classes, ˆ y ( x ) is the model’s prediction of y ∈ R K from x ∈ R M and is composed of a bijective function, h , a separable function, f k ( x )
= (cid:80) m f k,m ( x m ) , and the soft-max function, σ , such that ˆ y ( x ) =
σ ( f ( h ( x ))) .
If we are attempting to regularize the model to produce a single class label over D c , then y will correspond to a one-hot vector.
In general, however, y c may be a dense vector with elements y k .
The expected cross-entropy is expressed as
[Equation]
where p D c ( x | y ) is the probability density function over D c conditioned on y .
We can take advantage of the bijective transform to convert this expression into an expectation over the latent space, z , and latent domain, Z c , which allows us to write
[Equation]
where σ k is the k th element after the soft-max normalization and p Z c ( z | y ) is the independent probability density function in the latent space, e.g., p Z c ( z | y )
= (cid:81)
m p m ( z m ) .
We can then expand
[Equation]
The combination of the separable function and independent densities allows for an exact simpliﬁcation of the second term into
[Equation]
where we have overloaded Z m to correspond to the domain of the m th latent dimension and have approximated the integral via Monte Carlo integration with ˜ z m,g ˜ p
m ( z m ) and G draws.
The ﬁrst term on the right-hand side of Eq.
D3 does not enjoy the same exact simpliﬁcation.
However, it is possible to bound the term via Jensen’s Inequality by moving the logarithm into the sum over j or outside the expectation operator.
We consider the latter case where
[Equation]
Then, we expand the separable function, exchange the order of the sum over dimensions and the exponent, take advantage of the multiplicative integral simpliﬁcation (Eq. A5), approximate via
Monte Carlo integration, and utilize the log (cid:80) exp operator for stability: log
[Equation]
 
[Equation]
  E
[Equation]
c
[Equation]
=
[Equation]
exp(
[Equation]
m =
[Equation]
f
[Equation]
m )
[Equation]

 
 (D6)
= log  
K (cid:88)
j = (cid:90)
Z c d z
M (cid:89) m = exp( f
[Equation]
,m ( z m ))
M (cid:89) n = p n ( z n ) 
 =
log  
K (cid:88)
j =
M (cid:89) m = (cid:90)
Z m exp( f j,m ( z m ))
p m ( z m ) dz m 

≈ log  
K (cid:88)
j =
M (cid:89) m =
G (cid:88) g
= exp( f j,m (˜ z m,g ))

 − M log( G ) = log
K (cid:88) j
= exp
M (cid:88) m = log
G (cid:88) g
= exp( f j,m (˜ z m,g ))
− M log( G ) .
Finally, we can substitute Eq.
D4 and Eq.
D6 into Eq.
D1 to get
E D c [ CE (ˆ y ( x ) , y c )] ≤ − K (cid:88) k = y k
M (cid:88) m = (cid:90)
Z m f k,m ( z m ) p m ( z m ) dz m (D7) + log  
K (cid:88) j = M (cid:89) m = (cid:90) Z m exp( f j,m ( z m ))
p m ( z m ) dz m 

≈ − 1 G
K (cid:88) k
=
y k
M (cid:88) m =
G (cid:88) g = f k,m (˜ z m,g ) + log
K (cid:88) j
= exp
M (cid:88) m = log
G (cid:88) g
= exp( f j,m (˜ z m,g ))
− M log( G )
In the event that y c is a one-hot vector (at the c th element) this simpliﬁes to
[Equation]
where we reintroduced the condition over the class within the probability densities p
m ( z m | c ) for emphasis.