I T I S L IKELY T HAT Y OUR L OSS S HOULD BE A L IKELIHOOD
Mark Hamilton MIT, Microsoft markth@mit.edu Evan Shelhamer
MIT, Adobe Research shelhamer@adobe.com William Freeman MIT, Google billf@mit.edu
A BSTRACT
Many common loss functions such as mean-squared-error, cross-entropy, and reconstruction loss are unnecessarily rigid.
Under a probabilistic interpretation, these common losses correspond to distributions with ﬁxed shapes and scales.
We instead argue for optimizing full likelihoods that include parameters like the normal variance and softmax temperature.
Joint optimization of these “likelihood parameters” with model parameters can adaptively tune the scales and shapes of losses in addition to the strength of regularization.
We explore and systematically evaluate how to parameterize and apply likelihood parameters for robust modeling, outlier-detection, and re-calibration.
Additionally, we propose adaptively tuning L 2 and L 1 weights by ﬁtting the scale parameters of normal and Laplace priors and introduce more ﬂexible element-wise regularizers.
I NTRODUCTION
Choosing the right loss matters.
Many common losses arise from likelihoods, such as the squared error loss from the normal distribution , absolute error from the Laplace distribution, and the cross entropy loss from the softmax distribution.
The same is true of regularizers, where L 2 arises from a normal prior and L 1 from a Laplace prior.
Deriving losses from likelihoods recasts the problem as a choice of distribution which allows datadependent adaptation.
Standard losses and regularizers implicitly ﬁx key distribution parameters, limiting ﬂexibility.
For instance, the squared error corresponds to ﬁxing the normal variance at a constant.
The full normal likelihood retains its scale parameter and allows optimization over a parametrized set of distributions.
This work examines how to jointly optimize distribution and model parameters to select losses and regularizers that encourage generalization, calibration, and robustness to outliers.
We explore three key likelihoods: the normal, softmax, and the robust regression likelihood ρ of Barron (2019).
Additionally, we cast adaptive priors in the same light and introduce adaptive regularizers.
Our contributions:
We systematically survey and evaluate global, data, and predicted likelihood parameters and introduce a new self-tuning variant of the robust adaptive loss ρ 2.
We apply likelihood parameters to create new classes of robust models, outlier detectors, and re-calibrators.
We propose adaptive versions of L 1 and L 2 regularization using parameterized normal and Laplace priors on model parameters.
B ACKGROUND
Notation We consider a dataset D of points x
i and targets y i indexed by i ∈ { 1 , . . .
, N } .
Targets for regression are real numbers and targets for classiﬁcation are one-hot vectors.
The model f with parameters θ makes predictions ˆ
y i
=
f θ ( x ) .
A loss L (ˆ y, y ) measures the quality of the prediction given the target.
To learn model parameters we solve the following loss optimization:
[Equation] [Figure]
A likelihood L (ˆ y |
y, φ ) measures the quality of the prediction as a distribution over ˆ y given the target y and likelihood parameters φ .
We use the negative log-likelihood (cid:96) (NLL), and the likelihood interchangeably since both have the same optima.
We deﬁne the full likelihood optimization:
[Equation]
to jointly learn model and likelihood parameters.
“Full” indicates the inclusion of φ , which controls the distribution and induced NLL loss.
We focus on full likelihood optimization in this work.
We note that the target, y , is the only supervision needed to optimize model and likelihood parameters, θ and φ respectively.
Additionally, though the shape and scale varies with φ , reducing the error ˆ y − y always reduces the NLL for our distributions.
Distributions Under Investigation
This work considers the normal likelihood with variance σ (Bishop et al., 2006; Hastie et al., 2009), the softmax likelihood with temperature τ (Hinton et al., 2015), and the robust likelihood ρ (Barron, 2019) with shape α and scale σ that control the scale and shape of the likelihood.
The ﬁrst two are among the most common losses in machine learning, and the last loss provides an important illustration of a likelihood parameter that affects “shape” instead of “scale”.
We note that changing the scale and shape of the likelihood distribution is not “cheating” as there is a trade-off between uncertainty and credit.
Figure 1 shows how this trade-off affects the Normal and softmax distributions and their NLLs.
The normal likelihood has terms for the residual ˆ y − y and the variance σ as
[Equation]
with σ ∈ (0 , ∞ ) scaling the distribution.
The normal NLL can be written (cid:96) N = 12 σ 2 (ˆ y − y )
+log σ , after simplifying and omitting constants that do not affect minimization.
We recover the squared error by substituting σ = 1 .
The softmax deﬁnes a categorical distribution deﬁned by scores z for each class c as
[Equation]
with the temperature, τ ∈ (0 , ∞ ) , adjusting the entropy of the distribution.
We recover the classiﬁcation cross-entropy loss, − log p (ˆ y = y ) , by substituting τ = 1 in the respective NLL.
We state the gradients of these likelihoods with respect to their σ and τ in Section A of the supplement.
The robust loss ρ and its likelihood are
[Equation]
[Equation] with shape α ∈ [0 , ∞ ) , scale σ ∈ (0 , ∞ ) , and normalization function Z ( α ) .
This robust loss, ρ , has the interesting property that it generalizes several different loss functions commonly used in robust learning such as the L2 loss ( α = 2 ), pseudo-huber loss (Charbonnier et al., 1997)
( α = 1 ), Cauchy loss (Li et al., 2018) ( α = 0 ), Geman-McClure loss (Ganan & McClure, 1985), ( α = − 2 ), and Welsch (Dennis Jr & Welsch, 1978) loss ( alpha = −∞ ). Learning the shape parameter allows models to adapt the shape of their noise distribution.
R ELATED W ORK
Likelihood optimization follows from maximum likelihood estimation (Hastie et al., 2009; Bishop et al., 2006), yet is uncommon in practice for ﬁtting deep regressors and classiﬁers for discriminative tasks.
However Kendall & Gal (2017); Kendall et al.
; Barron (2019); Saxena et al.
(2019) optimize likelihood parameters to their advantage yet differ in their tasks, likelihoods, and parameterizations.
In this work we aim to systematically experiment, clarify usage, and encourage their wider adoption.
Early work on regressing means and variances (Nix & Weigend, 1994) had the key insight that optimizing the full likelihood can ﬁt these parameters and adapt the loss.
Some recent works use likelihoods for loss adaptation, and interpret their parameters as the uncertainty (Kendall & Gal, 2017; Kendall et al., 2018), robustness (Kendall & Gal, 2017; Barron, 2019;
Saxena et al., 2019), and curricula (Saxena et al., 2019) of losses.
MacKay & Mac Kay (2003) uses Bayesian evidence to select hyper-parameters and losses based on proper likelihood normalization.
Barron (2019) deﬁne a generalized robust regression loss, ρ , to jointly optimize the type and degree of robustness with global, data-independent, parameters.
Kendall & Gal (2017) predict variances for regression and classiﬁcation to handle data-dependent uncertainty.
Kendall et al.
(2018) balance multi-task loss weights by optimizing variances for regression and temperatures for classiﬁcation.
These global parameters depend on the task but not the data, and are interpreted as inherent task uncertainty.
Saxena et al.
(2019) deﬁne a differentiable curriculum for classiﬁcation by assigning each training point its own temperature.
These data parameters depend on the index of the data but not its value.
We compare these different likelihood parameterizations across tasks and distributions.
In the calibration literature, Guo et al.
(2017) have found that deep networks are often miscalibrated, but they can be re-calibrated by cross-validating the temperature of the softmax.
In this work we explore several generalizations of this concept.
Alternatively, Platt scaling (Platt, 1999) ﬁts a sigmoid regressor to model predictions to calibrate probabilities.
Kuleshov et al.
re-calibrate regressors by ﬁtting an Isotonic regressor to the empirical cumulative distribution function.
L IKELIHOOD P ARAMETER T YPES
We explore the space of likelihood parameter representations for model optimization and inference.
Though we note that some losses, like adversarial losses, are difﬁcult to represent as likelihoods, many different losses in the community have a natural probabilistic interpretation.
Often, these probabilistic interpretations can be parametrized in a variety of ways.
We explore two key axes of generality when building these loss functions: conditioning and dimensionality.
Conditioning We represent the likelihood parameters by three functional classes: global, data, and predicted.
Global parameters, φ = c , are independent of the data and model and deﬁne the same likelihood distribution for all points.
Data parameters, φ i , are conditioned on the index, i , of the data, x i , but not its value.
Every training point is assigned an independent likelihood parameter, φ i that deﬁne different likelihoods for each training point.
Predicted parameters, φ ( x ) =
g η ( x ) , are determined by a model, g , with parameters η (not to be confused with the task model parameters θ ).
Global and predicted parameters can be used during training and testing, but data parameters are only assigned to each training point and are undeﬁned for testing.
We show a simple example of predicted temperature in Figure 4, and an illustration of the parameter types in Figure 2.
We note that for certain global parameters like a learned Normal scale, changing the scale does not affect the optima, but does change the probabilistic interpretation.
This invariance has led many authors to drop the scale from their formulations.
However, when models can predict these scale parameters they can naturally remain calibrated in the presence of heteroskedasticity and outliers.
Additionally we note that for the shape parameter of the robust likelihood, ρ , changing global parameters does affect model ﬁtting.
Previous works have adapted a global softmax temperature for model distillation (Hinton et al., 2015) and recalibration (Guo et al., 2017).
Barron (2019) also experiments with global values of loss function shape and scale parameters.
The main work on Data parameters is that of Saxena et al.
(2019) who use these to learn a curriculum.
Model-based parameters appear in earlier work on regressing variance (Nix & Weigend, 1994), and more recent work by Kendall & Gal (2017).
Dimensionality The dimensionality, | φ | , of likelihood parameters can vary with the dimension of the task prediction, ˆ y .
For example, image regressors can use a single likelihood parameter for each image | φ | = 1 , RGB image channel | φ | =
C , or even every pixel | φ | =
W × H × C as in Figure 3.
These choices correspond to different likelihood distribution classes.
Dimensionality and Conditioning of likelihood parameters can interact.
For example, data parameters with | φ | =
W × H × C would result in N × W × H × C additional parameters, where N is the size of the dataset.
This can complicate implementations and slow down optimization due to disk I/O when their size exceeds memory.
Table 5 in the appendix contrasts the computational requirements of different likelihood parameter types.
The work of Barron (2019) explores both scalar and pixel-wise dimensionalities for his robust loss.
A PPLICATIONS
[Table] [Table] R OBUSTNESS AND O UTLIER D ETECTION
Data in the wild is noisy, and machine learning methods should be robust to noise, heteroskedasticity [Table] and corruption.
Unfortunately, models trained with the standard mean squared error (MSE) loss are highly susceptible to outliers [Table] and cannot naturally handle heteroskedasticity due to this loss’ ﬁxed variance (Huber, 2004).
Allowing models to predict and optimize their likelihood parameters allows models to generalize to these more complex settings.
More speciﬁcally, likelihood parameters naturally transform standard methods such as regressors, classiﬁers [Table] and manifold learners into robust variants without expensive outer-loop of model ﬁtting such as RANSAC (Fischler & Bolles, 1981) and Theil-Sen (Theil, 1992).
Figure 4 demonstrates this effect with a simple classiﬁcation dataset
[Table]
and we point readers to Figures 9 of the Supplement for similar examples for regression and manifold learning.
In certain datasets, even the assumption of Gaussianity is too restrictive and one must consider more robust and long-tailed distributions.
This has led many to investigate broader classes of likelihoods such as Generalized Linear Models (GLMs) (Nelder & Wedderburn, 1972) or the more recent general robust loss, ρ , of (Barron, 2019).
To systematically explore how likelihood parameter dimension and conditioning affect model robustness and quality, we reproduce Barron (2019)’s variational auto-encoding (Kingma & Ba, 2015) (VAE) experiments on faces from the CelebA dataset (Liu et al., 2015) in Table 1.
We explore learned data (Saxena et al., 2019) and model parameters in addition to Barron’s learned global parameters.
We also include two natural parameter dimensionalities: a single set of parameters for the whole image, and a set of parameters for each pixel and channel.
We ﬁnd that predicted parameters achieve the best performance while maintaining fast training time and a small memory footprint.
We also ﬁnd that pixel-wise learned parameters correlate with challenging areas of images and we visualize these parameters in Section D of the Appendix.
This experiment uses a 1 × 1 convolution on the last hidden layer of the decoder as a likelihood parameter model and has the same resolution as the output. The low and high dimensional losses use the same convolutional regressor, but the 1 dimensional case averages over pixels. In the high dimensional case, the output has three channels (for RGB), with six channels total for shape and scale regression. We use the same non-linearities to constrain the shape and scale outputs to reasonable ranges as in (Barron, 2019).
More speciﬁcally, we use an afﬁne sigmoid to keep the shape α ∈ [0 , 3] and the softplus to keep scale c ∈ [10 − 8 , ∞ ) . Table 1 gives the results of evaluating each method by MSE on the validation set, while training each method with their respective loss parameters. Data parameter optimization uses Tensorﬂow’s implementation of sparse RMSProp (Tieleman & Hinton, 2012).
We also inherit weight decay (cid:107) φ (cid:107) 22 , gradient clipping ∇ φ / (cid:107)∇ φ (cid:107) 22 , and learning rate scaling α φ = α · m for learning rate α and multiplier m from Barron (2019).
The robustness we see in our VAE experiments stems from the fact that likelihood parameter prediction gives models a direct channel to express their “uncertainty” for each data-point with respect to the task.
This allows models to naturally down-weight and clean outliers from the dataset which can improve model robustness.
Consequently, one can harness this effect to create outlier detectors from any underlying model architecture by using learned scales or temperatures as an outlier score function.
Furthermore, predicted likelihood parameters allow these methods to detect outliers in unseen data.
In Figure 5 we show how auditing temperature or noise parameters can help practitioners spot erroneous labels and poor quality examples. In particular, the model-parameterized temperatures of
[Table] an image classiﬁer (trained using the setup of 5.3) correlates strongly with blurry, dark [Table] and difﬁcult examples on the Street View House Number (SVHN) dataset.
We use this approach to create simple outlier detection algorithms by considering deep (AE+S) and linear (PCA+S) auto-encoders (Kramer, 1991) with data-conditioned scale parameters as outlier scores. We evaluate this approach on tabular datasets using deep and linear auto-encoders with model-parameterized scales.
In Table 2 we quantitatively demonstrate the quality of these simple likelihood parameter approaches across 22 datasets from the Outlier Detection Datasets (ODDS), a standard outlier detection benchmark (Rayana, 2016).
The ODDS benchmark supplies ground truth outlier labels for each dataset [Table] which allows one to treat outlier detection as an unsupervised classiﬁcation problem.
We compare against a variety of established outlier detection approaches included in the pyOD (Zhao et al., 2019) framework including: One-Class SVMs (OCSVM) (Sch¨olkopf et al., 2000), Local Outlier Fraction (LOF)
(Breunig et al., 2000), Angle Based Outlier Detection (ABOD) (Kriegel et al., 2008), Feature Bagging (FB) (Lazarevic & Kumar, 2005), Auto Encoder Distance (AE) (Aggarwal, 2015), K-Nearest Neighbors (KNN) (Ramaswamy et al., 2000; Angiulli & Pizzuti, 2002), Copula Based Outlier Detection (COPOD) (Li et al., 2020), Variational AutoEncoders (VAE) (Kingma & Welling, 2013), Minimum Covariance Determinants with Mahlanohbis Distance (MCD) (Rousseeuw & Driessen, 1999;
Hardin & Rocke, 2004), Histogram-based Outlier Scores (HBOS) (Goldstein & Dengel, 2012), Principal Component Analysis (PCA)
(Shyu et al., 2003), Isolation Forests (IF) (Liu et al., 2008; 2012), and the Clustering-Based Local Outlier Factor (CBLOF) (He et al., 2003).
Our predicted scale auto-encoders use PyTorch’s layers API (Paszke et al., 2019) with rectiﬁed linear unit (ReLU) activations for deep auto-encoders and Glorot uniform initialization (Dahl et al., 2013; Glorot & Bengio, 2010) for all layers. We use Adam (Kingma & Ba, 2015) with a learning rate of 0005 for 4000 steps with 20% dropout before the code space.
We follow ODDS guidelines and standard scale the data prior to ﬁtting. Our methods (PCA+S and AE+S) use a similar principle as isolation-based approaches that determine outliers based on how difﬁcult they are to model.
In existing approaches, outliers inﬂuence and skew the isolation model which causes the model to exhibit less conﬁdence on the whole.
This hurts a model’s ability to distinguish between inliers and outliers.
In contrast, our approach allows the underlying model to down-weight outliers.
This yields a more consistent model with a clearer decision boundary between outliers and inliers as shown in Figure 4.
As a future direction of investigation we note that our approach is model-architecture agnostic, and can be combined with domain-speciﬁc architectures to create outlier detection methods tailored to images, text, and audio.
5. A DAPTIVE R EGULARIZATION WITH P RIOR P ARAMETERS
In addition to optimizing the shape and scale of the likelihood distribution of the model output, we can use the same approach to optimize the prior distribution of the model parameters.
More speciﬁcally, we propose adaptive regularizers for a model’s parameters, θ .
This approach optimizes the distribution parameters of the prior, φ prior , to naturally tune the degree of regularization.
In particular, the Normal (Ridge, L2) and Laplace (LASSO, L1) priors, with scale parameters σ and b , regularize model parameters for small magnitude and sparsity respectively (Hastie et al., 2009).
The degree of regularization, λ ∈ [0 , ∞ ) , is conventionally a hyperparameter of the regularized loss function:
[Equation]
We note that we cannot choose λ by direct minimization because it admits a trivial minimum at λ = 0 .
In the linear case, one can select this weight efﬁciently using Least Angle Regression (Efron et al., 2004).
However, in general λ is usually learned through expensive cross validation methods.
Instead, we retain the prior with its scale parameter, and jointly optimize over the full likelihood:
[Equation]
This approach, the Dynamic Lasso (D-LASSO), admits no trivial solution for the prior parameter b , and must balance the effective regularization strength, 1 b , with the normalization factor, log b .
D-LASSO selects the degree of regularization by gradient descent, rather than expensive black-box search.
In Figure 7 (left) and (middle) we show that this approach, and its Ridge equivalent, yield ideal settings of the regularization strength on a suite of synthetic regression problems.
Figure further we replace parameter θ j locally adapt regularization each model weight M -Lasso). This consistently outperforms any global setting of the regularization strength and shields important weights from undue shrinkage our experiments we use 500 samples of 500 dimensional normal distributions mapped through linear functions with additive gaussian noise.
Linear transformations use LASSO experiments sparse transformations
We use tensorﬂow’s Our approach of learning regularizer scale parameters can be viewed naturally through the lens of hierarchical priors (Gelman et al., 2013). More speciﬁcally this approach is implicitly performing maximum a posteriori (MAP) inference on the prior’s scale with respect to a uniform prior on that parameter.
We note that though these methods for hyperparameter selection are common in the Bayesian literature, they are not widely used in practice in the deep learning community. This work aims to bring these parameters back within the scope of deep learning where they can be easily expanded to more ﬂexible forms such as our introduced Multi-Lasso. 5. R E - CALIBRATION
The work of (Guo et al., 2017) shows that modern networks are accurate, yet systematically overconﬁdent, a phenomenon called mis-calibration.
We investigate the role of optimizing likelihood parameters to re-calibrate models.
More speciﬁcally, we can ﬁt likelihood parameter regressors on a validation set to modify an existing model’s conﬁdence to better align with the validation set.
This approach is a generalization of Guo et al.
(2017)’s Temperature Scaling method, which we refer to as Global Scaling (GS) for notational consistency.
Global Scaling re-calibrates classiﬁers with a learned global parameter, τ in the loss function: σ ( (cid:126)x, τ ) .
Fitting model-conditioned likelihood parameters to a validation set deﬁnes a broad class of recalibration strategies.
From these we introduce three new re-calibration methods.
Linear Scaling (LS) learns a linear mapping, l , to transform logits to a softmax temperature: σ ( (cid:126)x, l ( (cid:126)x )) .
Linear
Feature Scaling (LFS) learns a linear mapping, l , to transform the features prior to the logits, (cid:126)f , to a softmax temperature: σ ( (cid:126)x, l ( (cid:126)f )) .
Finally, we introduce Deep Scaling (DS) for regressors which learns a nonlinear network, N , to transform features, (cid:126)f , into a temperature: σ ( (cid:126)x, N ( (cid:126)f )) .
In Table 3 we compare our recalibration approaches to the previous state of the art: Global Scaling.
We note that (Guo et al., 2017) have already shown that Global Scaling outperform Bayesian Binning into Quantiles (Naeini et al., 2015), Histogram binning (Zadrozny & Elkan, 2001), and Isotonic
Regression.
We recalibrate both ResNet
(He et al., 2016) and DenseNet121 (Huang et al., 2017) on a variety of vision datasets.
We measure classiﬁer miscalibration using the Expected Calibration
Error (ECE) (Guo et al., 2017) to align with prior art.
We additionally evaluate Isotonic recalibration,
Platt Scaling (Platt, 1999), and Vector Scaling (VS) (Guo et al., 2017), which learns a vector, (cid:126)v , to re-weight logits: σ ( (cid:126)v(cid:126)x, 1) .
LS and LFS tend to outperform other approaches like GS and VS, which demonstrates that richer likelihood parametrizations can improve calibration akin to how richer models can improve prediction. Our experiments leverage Tensorﬂow’s Dataset APIs that include the SVHN, (Netzer et al., 2011),
ImageNet (Deng et al., 2009), CIFAR-100, CIFAR-10 (Krizhevsky, 2009) datasets.
We use Keras implementations of DenseNet-121 (Huang et al., 2017) and ResNet-50 (He et al., 2016) with default initializations.
For optimization we use Adam with lr = 0 .
0001 , β 1 = . 9 , β 2 = . 99 (Kingma & Ba, 2015) and train for 300 epoch with a batch size of 512. For recalibrating regressors, we compare against the previous state of the art, Kuleshov et al.
(2018), who use an Isotonic regressor to correct a regressors’ conﬁdence.
We use the same experimental setting as Kuleshov et al. (2018) including the UCI datasets (Dua & Graff, 2017), and regressor calibration metric (CAL).
Table 4 shows that our approaches can outperform this baseline as well as the regression equivalent of Global Scaling.
Inputs and targets are scaled to unit norm and variance prior to ﬁtting for all regression experiments and missing values are imputed using scikit-learn’s “SimpleImputer” (Pedregosa et al., 2011).
Experiments utilize Keras’ layers API with two hidden rectiﬁed linear unit (ReLU) layers, Glorot uniform initialization (Dahl et al., 2013; Glorot & Bengio, 2010) and Adam optimization with lr = 0 .
001 for 3000 steps without minibatching.
[Table] [Table]
E XPERIMENTAL D ETAILS
We run all experiments on Ubuntu 16.04 Azure Standard NV24 virtual machines (24 CPUs, 224 Gb memory, and 4 × M60 GPUs) with Tensorﬂow 1.15 (Abadi et al., 2015) and PyTorch 1.17 (Paszke et al., 2019).
Many likelihood parameters have constrained domains, such as the normal variance σ ∈ [0 , ∞ ) .
To evade the complexity of constrained optimization, we deﬁne unconstrained parameters φ u and choose a transformation t ( · ) with inverse t − 1 ( · ) to map to and from the constrained φ .
For positivity, exp / log parameterization is standard (Kendall & Gal, 2017; Kendall et al., 2018;
Saxena et al., 2019).
However, this parameterization can lead to instabilities and we use the softplus, s + ( x )
{+= log(
+}
+ exp( x )) , instead.
Shifting the softplus, s + c ( x ) =
( ln (1 + e x ) + c ) /
( ln (2) + c ) , further improves stability and we explore this effect in Figure 12 of the Appendix.
We use an afﬁne softplus s + . 01 and s + . 2 respectively for adaptive scales and temperatures respectively.
The one exception is adaptive regularizer scales where we found exp led to faster convergence.
For the constrained interval [ a, b ] we use afﬁne transformations of the sigmoid s ( x ) = 1 1+exp( − x ) (Barron, 2019).
We initialize likelihood parameter biases to settings that yield MSE and Cross Entropy ( σ = τ = 1 ).
C ONCLUSION
Optimizing the full likelihood can improve model quality by adapting losses and regularizers.
Full likelihoods are agnostic to the architecture, optimizer, and task, which makes them simple substitutes for standard losses.
Global, data, and predicted likelihood parameters offer different degrees of expressivity and efﬁciency.
In particular, predicted parameters adapt the likelihood to each data point during training and testing without signiﬁcant time and space overhead.
By including these parameters in a loss function one can improve a model’s robustness and generalization ability and create new classes of outlier detectors and recalibrators that outperform baselines.
More generally, we hope this work encourages joint optimization of model and likelihood parameters, and argue it is likely that your loss should be a likelihood.