Boosting the Performance of Generic Deep Neural Network Frameworks with Log-supermodular CRFs
Abstract
Historically, conditional random fields (CRFs) were popular tools in a varietyof application areas from computer vision to natural language processing, butdue to their higher computational cost and weaker practical performance, theyhave, in many situations, fallen out of favor and been replaced by end-to-end deepneural network (DNN) solutions.
More recently, combined DNN-CRF approacheshave been considered, but their speed and practical performance still falls shortof the best performing pure DNN solutions.
In this work, we present a genericcombined approach in which a log-supermodular CRF acts as a regularizer toencourage similarity between outputs in a structured prediction task.
We show thatthis combined approach is widely applicable, practical (it incurs only a moderateoverhead on top of the base DNN solution) and, in some cases, it can rival carefullyengineered pure DNN solutions for the same structured prediction task.
Introduction
A wide range of machine learning applications can be cast as structured prediction problems, jointprediction problems in which the aim is to produce a vector of correlated outputs rather than a singleclass label, e g., semantic image segmentation, stereo matching, etc.
Historically, conditional randomfields (CRFs) were popular for this class of problems as they (1) allow modeling of the underlyingproblem structure, (2) can incorporate domain knowledge and (3) admit efficient (approximate)inference in certain settings.
More recently, however, increases in data set and model sizes have made
CRF-based approaches less practical.
Instead, deep neural networks (DNNs) that can be trained atscale by taking advantage of accelerators like GPUs and TPUs have become increasingly popular.
While deep neural networks yield powerful predictive models, they can be sample inefficient [26]and may fail to produce solutions that respect natural constraints required by the structured output,e.g., if the output is a combinatorial object.
In practice, the structured outputs of DNNs can lacksimple smoothness properties that one would expect in a given domain: (1) in stereo matching,neighboring pixels likely have similar depths, (2) in optical flow estimation, neighboring pixels likelyhave similar depths, (3) in activity recognition in video, objects/activities are likely to persist betweenframes, (4) in motion/trajectory prediction tasks, objects in motion tend to stay in motion in the samedirection, (5) in image colorization, neighboring pixels likely have similar colors, (6) in segmentation,neighboring pixels are more likely to be part of the same class, and (7) in certain statistical physicsapplications, neighboring particles tend to behave similarly.
As a result, the ability to incorporate thistype of soft knowledge could improve the training of DNN models and smooth the structured output.
Motivated by existing approaches that combine DNNs with CRFs, such as structured variationalautoencoders [22] and models for stereo depth estimation [24, 31], we propose a generic hybridmodel, which takes an existing DNN, combines it with a CRF, and then jointly trains the combinedmodel, for the above smoothing task.
In order to accomplish the smoothing, our CRFs will employ log-supermodular potentials.
This class of potential functions belongs to the family of so-calledattractive graphical models [54] as the potential functions encourage agreement between the variablesover which they operate.
As such, we dub our approach attractive smoothing .
Our aim is to show that(1) the training and prediction overhead of such a hybrid model can be kept low and (2) a sufficientlyexpressive CRF can act like a regularizer for the DNNs by smoothing the outputs as described above.
Our approach applies in both discrete and continuous settings and, to the best of our knowledge, ourapproach represents a novel, practical application of continuous log-supermodular CRFs.
Via a series of experiments, we show that attractive smoothing can be applied to yield both qualitativeand quantitative performance improvements with only a modest overhead compared to the original
DNN in a range of applications (stereo matching, semantic segmentation, and image colorization).
Our experiments show that even a simple CRF can smooth local neighborhoods and encourage localconsistency.
In our experiments, the hybrid approach always resulted in a performance improvement,sometimes dramatically, over the pure DNN architecture.
In particular, we observed that evencomparatively older architectures can be competitive against (or outperform) more recent architecturessimply with the addition of an attractive CRF.
Preliminaries
We will use CRFs to represent a joint probability distribution over a structured output y given dataobservations x that factorizes as a product of nonnegative potential functions over a given graphstructure.
Specifically, given a graph G =
( V, E ) and ϕ i ( ·| x ) :
Y → R ≥ 0 and ϕ ij ( · , ·| x ) :
Y × Y → 55 R ≥ 0 , a pairwise CRF models p ( y | x ) as
[Equation] [Equation]
(cid:89)
[Equation] [Equation]
, where Z ( x ) is the normalizing constant that ensures that p ( ·| x ) is a probability distribution.
The graph structure and potential functions are problem dependent – they should be chosen to reflectthe structure, correlations, and constraints of the output.
In particular, we will be interested inlog-supermodular (LSM) potential functions as they encourage agreement between the variablesupon which they operate.
Formally, a function f : D → R ≥ 0 for D ⊆ R n is log-supermodular iff ( x ) f ( y )
≤ f ( x ∧ y )
f ( x ∨ y ) , for all x, y ∈ D , where for all i ∈ { 1 , . . . , n } , ( x ∧ y ) i =
min( x i , y i )and ( x ∨ y )
i = max( x i , y i ) .
Note that D should be closed under these operations in order for theabove definition to make sense, e.g., D = { 0 , 1 } n or D = [0 , 1] n .
If the inequality is reversed fis said to be log-submodular (submodular and supermodular functions satisfy a similar inequalitywith products replaced by sums).
A twice continuously differentiable function f : D → R ≥ 0 islog-supermodular on D if and only if for all i ̸ = j ∈ { 1 , . . . , n } and x ∈ D , ∂ 2 log( f ) ∂x i ∂x j | x ∈ D ≥ 0 , i.e.,the off-diagonal elements of the Hessian matrix of log( f ( · )) are nonnegative over the domain [51].
From the definition, products of log-supermodular functions are log-supermodular.
So, if all of thepotential functions are log-supermodular, so is p ( y | X ) .
Log-supermodular (log-submodular) functions arise in a variety of applications and represent classesof functions for which global maxima (minima) can be found efficiently in both discrete and continuous settings [14, 47, 21, 1].
Discrete log-supermodular/log-submodular functions also have arich history in the study of CRFs in both theory and practice, e.g., [19, 5, 25].
Continuous logsupermodular CRFs enjoy similar theoretical properties to their discrete counterparts [41, 42, 2].
However, despite the wide use of discrete log-supermodular/log-submodular models in real-worldsettings, e.g., in a variety of computer vision tasks, continuous log-supermodular CRFs have notreceived much attention in practical applications.
Attractive Smoothing via Log-supermodular CRFs
The structured prediction problems considered here are supervised learning problems in which we aregiven data observations x (1) , . . .
, x ( M ) ∈ R d and corresponding structured outputs y (1) , . .
, y ( M ) ∈
Y .
The aim is to learn a model that can predict the structured outputs for novel data observations.
In order to take advantage of the predictive power of deep neural networks in combination with themodeling power of CRFs for this task, we propose the simple end-to-end model shown in Figure 1.
[Figure]
Net
CRF Inference Layer
Loss y
Optimizer
Init.
ˆ y
Node Pot.
Edge Pot.
Minimize BFE [ ] , ,       , ,    ( ) | , i i y w v  ( ) | , ij i j y , y w v  ( ) d f x v = ( ) p f x w =
In this architecture, two neural networks, a discriminator NN and a parameter NN, are trained toconstruct the parameters of a CRF given a data observation x .
Inference is then performed over the
CRF in order to produce the final prediction, ˆ y , which is then fed into a loss function.
Specifically, thediscriminator network produces a vector of outputs v and the parameter network produces a vector ofoutputs w and the CRF models the joint probability of the structured output y given v and w .
In applications, we will think of the discriminator net as a given DNN solution for the structuredprediction task whose outputs we want to smooth and the parameter net will control the strength ofthe relationship between neighboring variables
y i and y j .
As a motivating example, consider a taskin which the input is a video and the aim is to determine whether or not each frame of the videocontains an object from a known finite set, F . Consider a DNN that takes as input the i th frame ofvideo and produces a vector v ( i ) ∈ { 0 , 1 } | F | , where v ( i ) = 1 indicates that the DNN predicts thatobject f is in frame i .
Applying this DNN for each frame of video generates a sequence of vectorsand stacking the results produces a vector v , which we can treat as the output of the discriminator netin our architecture.
If an object appears in frame i , it is likely to occur in frame
i + 1 as well.
We canconstruct a chain-structured CRF, with one binary random variable, y ( t ) f , for each f ∈ F at each timeslice t , whose potentials at time slice t are given by
[Equation]
where a and c ≥ 0 are the outputs of the parameter net (indexed by f ∈ F and time steps or pairs oftime steps).
The nonnegativity constraint on c ensures that each potential function is log-supermodular.
The effect of these potentials is to encourage the smoothness property over the sequence of videoframes for the object detection task.
More generally, when applying attractive smoothing for structured objects y ∈ S for some finiteset S ⊂ R , the univariate potential functions could be arbitrary (as univariate functions are alwayslog-supermodular) and the pairwise potentials could be of the form exp( g ( y i , y j | v, w )) , whereg ( y i , y j | v, w ) is a supermodular function of y i and y j , i.e., g ( y i , y j | v, w ) + g ( y ′
i , y ′ j | v, w ) ≤ g ( y i ∧y ′
i , y j ∧
y ′
j | v, w ) + g ( y i ∨ y
′ i , y j ∨ y ′
j | v, w ) for all y i , y j ,
y ′
i , y ′ j ∈ S .
Note that in the most generalcase, we assume that the potential function values for each input/output pair are parameters of the
CRF.
The linear constraint on the pairwise potential functions could be added to the machine learningobjective (discussed below) and enforced, with additional overhead, via projected gradient methods.
3.1 The Continuous Case
In the case of continuous models, similarities between random variables are often modeled using amultivariate Gaussian distribution.
There are two primary reasons for this: (1) multivariate Gaussiansrepresent a simple, well-understood parametric family and (2) Gaussian CRFs admit exact MAPinference via matrix inversion.
While this can be slow in practice for large models, in specialsub-classes of Gaussian models, e.g., walk-summable Gaussian graphical models [33, 43], exact
MAP inference can be done with Gaussian belief propagation, or similar approaches, without theexpense of a full matrix inversion.
Further, GPU versions of approximate message-passing algorithms,mean-field methods, or approximate variational inference methods, e.g., [55], can be used to makeinference extremely practical even in large-scale models.
As an example of an approach based on walk-summable models, consider potentials of the form
[Equation]
i.e., v i is the mean parameter of a Gaussian potential in the CRF, which is given by the discriminatornet, and the parameterized pairwise potentials are of the form
[Equation]
The parameters, w ≥ 0 , are given as the output of the parameter NN.
Note that the nonnegativityconstraint on w is sufficient to ensure that the joint distribution is a welk-summable Gaussian.
Thismodel is a generalization of the modeling approach taken by Liu et al.
[31] and others.
The Gaussian CRF acts as a regularizer/smoother to encourage (or not) neighboring pixels to takesimilar values depending on how the parameter net selects the w parameters.
While the CRF in thisexample admits efficient inference, the modeling power of the potential functions is limited by the
Gaussian requirement.
Note that this model is log-supermodular by construction as the coefficientsof the y i
y j terms is equal to w ij , which is constrained to be nonnegative.
If we relax the Gaussianrequirement, the resulting log-quadratic model may not be integrable.
However, in many practicalapplications, the structured outputs y come from compact subset of R n of the form [ a, b ] n , andwe can weaken the Gaussian requirement to only requiring log-supermodularity in such a setting.
Specifically, consider potential functions of the form
[Equation]
where v is the output of discriminator net and w = ( a, b, c ) is the output of parameter net.
Fromthe second order condition for log-supermodularity, imposing the constraint that c ≥ 0 is enough toensure that each potential, and hence the entire model, is log-supermodular.
The log-supermodular model above contains the Gaussian model as a special case, but it need notproduce a joint Gaussian model (it could correspond to a log-concave or log-convex distribution, butin general, it will be neither as the a ’s can be either be positive or negative).
We note that we willrestrict log-supermodular models in the continuous case to log-quadratic functions for computationalreasons as well as to avoid potential overfitting from much more flexible models.
Despite thislimitation, we are unaware of other works that take a similar approach in the continuous case.
End-to-end Training of Hybrid Models
Fitting the combined NN+CRF model to data requires selecting an appropriate loss function, e.g.,negative conditional log-likelihood, squared error, etc., and performing gradient descent to minimizethe loss.
Different loss functions will result in different updates.
Example loss functions that arepopular in combined frameworks include the negative conditional log-likelihood (NLL), squared loss(SL), and the Huber loss (HL).
No matter which loss function is selected, (approximate) inferencewill need to be performed on the CRF as part of the gradient update: For the NLL, this can bereduced to performing marginal inference, and for SL, HL, or similar discriminative losses, weneed to differentiate the arg max y p ( y | x ) with respect to the NN parameters, which can be done bycomputing the maximum of p ( ·| x ) using MAP inference and then applying the chain rule (constructpartial derivatives with respect to the CRF parameters and then use backpropagation).
Approximate MAP inference in CRFs with continuous random variables can be performed usinga variety of different approaches, e.g., mean-field variational inference, particle belief propagation[20, 27], or more general approximate variational inference strategies [17, 55].
Here we use theapproximate variational methods, e.g., mean-field or the Bethe Free Energy (BFE), as they can beeasily and efficiently implemented as a recurrent NN layer to seamlessly connect with the target
NN output.
In both cases, we can think of the resulting (approximate) variational objective as anoptimization problem over mean parameters, µ , and covariances, Σ .
In the log-supermodular case, we could apply polynomial-time approximation schemes to estimatethe MAP solution, e.g., [53, 1], but these strategies are likely to be too expensive, especially if wewant to guarantee high accuracy.
Practically speaking, the approximate variational methods areprobably still preferred in practice, e.g., mean-field or the MAP inference approach of Xiong et al.[55], as these approaches are embarrassingly parallelizable and easy to implement as a computation layer in an ML framework like Tensorflow and Pytorch that can be seamlessly appended at the endof the original DNN (the overall computational overhead was around 10% in our experiments, see
Table 4).
The approximate variational methods are not guaranteed to return a global optimum, butthe polynomial-time approximation schemes could be used to pick a good starting point for theapproximate variational methods, e.g., by discretizing the domain and solving a simpler, approximate,discrete log-supermodular problem, in order to avoid poor local optima (we do not adopt this approachas it does not seem to be necessary to obtain performance improvements in practice).
Related Work
Historically, models based purely on CRFs have been used extensively for both stereo/monoculardepth estimation, e.g., [24, 29, 32, 31, 45, 46, 49], as well as semantic image segmentation, e.g.,[39, 38, 58].
The key challenge when applying CRFs for these problems is that fitting the CRF modeland performing inference is typically intractable, which necessitates approximate inference, e.g.,mean-field methods or particle belief propagation [20].
Prediction using the approximate methodscan also be slow or inaccurate, e.g., particle BP may take a long time to converge (if it does atall).
In addition, superpixels or subregions are often used to make the inference in these modelsscalable.
Finally, features are typically hand-constructed, which often reduces practical performance.
Motivated by the above limitations of pure CRF solutions, frameworks have sought to combine themwith DNNs, e.g., for stereo matching [31, 24], for image segmentation [30, 61, 62, 9, 48, 28, 40]and for image colorization Messaoud et al.
[37], or abandon CRFs altogether.
While this list ofhybrid approaches is not exhaustive for vision applications, it is illustrative of the limitations ofmany of the existing combined approaches.
The end-to-end framework we will explore in the nextsection differs from existing DNN+CRF approaches in key ways: (1) we use a more general classof log-supermodular potentials, (2) our GPU implementation of the approximate inference routinesmakes our approach scalable even at the pixel level for vision tasks, (3) discretization is not used toreduce continuous inference to discrete inference, and (4) our models are trained with discriminativelosses as opposed to minimizing the negative log-likelihood.
Similarly, there has been significant work on structured prediction that combines DNNs with CRFs.
While our focus here is on log-supermodular CRFs, a number of general approaches have beenproposed, and we highlight some of the most related here.
Belanger et al.
[4] consider a generalframework for continuous energy minimization (SPEN).
In contrast to the approach here whichcombines log-supermodular CRFs with DNNs, they consider a single black-box NN energy function(which can easily lead to overfitting in practice).
Graber and Schwing [18] proposed an alternative to
SPEN that allows problem structure to be incorporated to the learning process as part of a generalizedstructured SVM framework.
The approach here is much simpler (and more targeted) and consequentlyinference is less time consuming (hence more scalable).
Experimental Evaluation
In this section, we evaluate the hybrid framework for attractive smoothing using existing DNNmodels developed for a variety of different machine learning applications.
Below we describe theexperimental setup in detail and then present qualitative and quantitative results for each of theapplication domains.
All models were implemented using Tensorflow 2.2 and run on a 3 × NVIDIA
Tesla V100 GPU.
All source code along with learned model weights will be released on GitHub uponacceptance (for now they can be accessed via the supplementary material).
In each application, weuse the following skeleton.
Discriminator Net: a DNN that takes in the problem input and produces a structured output, y .
Parameter Net: a simple stacked hourglass convolutional net that takes the problem input and
outputs the potential function parameters.
See
Table 3 in the appendix for details.
217 CRF and Potentials: Grid CRF of size H ×
W , (which gives HW nodes and 2 HW edges), where
H and W are the height and width of input image(s) for the DNN.
We evaluate two different types ofhybrid models: the typical Gaussian potentials from equations (1) and (2), which we denote as SG,and the log-supermodular potentials from (3) and (4), which we denote as LSM.
[Table]
[Table]
Marginal inference: Approximate inference is done via gradient ascent on the mean-field lower
[Table]
bound using the GPU scheme of Xiong et al.
[55] with learning rate η = 0 .
The MAP assignment
[Table]
is computed from the joint distribution returned by the inference routine.
[Table]
6.1 Stereo Matching
Given a pair of input images (left, right), the goal of stereo matching is to compute the disparity d foreach pixel in the reference image (the left image in this experiment).
Given their speed and accuracy,end-to-end DNN solutions have become the most popular approach for stereo matching, e.g., GC-Net[23], PSM-Net [8], GA-Net [57], CSPNs [11], LEAStereo [13], CSPNs [12], HITNET [50], etc.
We selected several architectures for evaluation, GC-Net, PSM-Net, and LEAStereo as many recentarchitectures have been built from components of the first two and LEAStereo was near the top ofthe KITTI 2015 leaderboard at time of submission:
GC-Net was the first architecture to learn a costaggregation function (as opposed to hand-crafted cost aggregation functions) and also suggestedusing a differentiable soft-max while PSM-Net minimizes multiple losses during training (two arediverted from intermediate layers to prevent skewing of information).
Model Details
The parameter net only takes the left image as input and outputs the potential parameters w for SG and a, b, c for LSM.
The parameters c and w control how similar two neighbouringpixel’s disparity should be.
Intuitively, in SG, w , should be large for pixels in the same object surfaceand smaller for boundary pixels.
All models are fit by minimizing the Huber loss:
[Equation]
We set δ = 1 to match the setting of Chang and Chen [8].
For consistency, all models were trainedfrom scratch with the Adam optimizer ( β 1 = 0 .
9 , β 2 = 0 .
999 ), learning rate set to 0.0001, and batchsize 6.
For each training batch, input images were randomly cropped to a dimension of 256 × 512 .
The maximum disparity ( D ) is set to 192.
Datasets We evaluate the proposed approach using the Scene Flow [34] and Virtual KITTI 2 [6],synthetic data sets are often used to pretrain DNNs that can be fine-tuned on much smaller real-worlddata sets such as KITTI [36, 35].
Detailed data set descriptions can be found in Appendix B. Thetraining epochs for each data set are: 30 for Sceneflow and 25 for Virtual KITTI 2.
Evaluations weredone on full-sized images in the respective test sets.
Results For the experimental evaluation, we compare pure DNN architectures with combined
DNN+CRF architectures on the data sets discussed above.
We use the mean absolute error betweenthe predicted disparity and the ground truth (MAE), sometimes referred to as end point error (EPE),and k -pixel threshold error percentage, i.e., the fraction of pixels whose absolute error is larger thank pixels, as our evaluation metrics.
The evaluation results are computed on all regions, both occludedand non-occluded areas, except if the pixel does not have a valid ground truth or is greater than afixed maximum disparity setting ( D ).
Our aim is to investigate whether or not the combined approach
(1) results in significant performance improvements with respect to MAE and 1 pixel error over thepure NN approaches and (2) is efficient for both training and prediction.
The CRFs add an additional half-million or so parameters to the model but does not seem to result ina significant increase in prediction time despite the 10-20% increase in the number of parameters,e.g., GC-Net has 2.9M while GC-Net+CRF has 3.4M while the prediction time on the Sceneflow testset only increases by about .04s when the CRF is added.
See Appendix C for additional speed/sizedetails.
We expect that similar observations would hold in other applications as long as the number ofparameters of the parameter net and the CRF scale moderately with the dimension of y .
Table 1 presents the test set performance of the different models both with and without CRFs on thedata sets discussed above.
For comparison, we also include the published performance of GA-Net[57] on the Sceneflow data set (it was trained using the same loss function).
We have the followingobservations.
(1) Overall, the combined approaches outperform their respective pure approaches interms of both MAE and 1 pixel error.
On SceneFlow, the most significant performance improvementsoccurred for PSM-Net, both metrics were reduced between with LSM 40%-50%.
And the resultingmodel had the best overall performance on Sceneflow.
On Virtual KITTI2, adding LSM to GC-Netachieved the biggest performance improvements and also resulted in the best performing model on thisdata set.
(2) Even the comparatively older architectures GC-Net and PSM-Net perform comparably orbetter than the more recent LEAStereo when a LSM CRF is used.
PSM-Net (LSM) achieves the best
Sceneflow performance while GC-Net (LSM) achieved the best Virtual KITTI2 performance.
(3) Theperformance improvement obtained by hybrid approaches on Sceneflow applied to the LEAStereomodel are minimal.
We suspect that this is because the model search that was led to the construction of
LEAStereo was tailored to this data set.
However, on Virtual KITTI2, LEAStereo’s base performanceis only slightly better than GC-NET and PSM-Net, but the performance is improved significantly byadding a CRF.
The LSM CRF always outperforms the SG CRF, sometimes significantly.
While
SG is a special case of LSM, we observed that LSM performed better on both train and test data.
This suggests that the added flexibility of LSM over SG does not result in significant overfitting.
Additional quantitative results can be found in the supplementary material.
A qualitative comparison over the PSM-Net models can be found in Figure 2.
As expected, theaddition of the CRF appears to result in smoother local neighborhoods compared to the base modeland better predictions on regions with fine details and thin areas, e.g., the spokes of wheels and theleaves of plants.
Comparing SG with LSM, we see that SG tends to oversmooth, especially in regionswith a lot of fine details, as compared to LSM – this seems to be the primary difference between thetwo models.
Additional qualitative results can be found in the supplementary materials.
6.2 Image Colorization
Next, we considered the problem of image colorization: given a grayscale image as input, the modelneeds to generate a full color image.
Colorization is an ill-posed problem, requiring mapping a real-valued luminance image to a three-dimensional color-valued one, for which a unique solutionmay not exist.
Many early DNN models used an encode-decoder, FCN architecture, [59, 3, 60], withsome using classification loss, e.g., Zhang et al.
[59], while others used smoothed ℓ 1 loss, e.g., Zhanget al.
[60] or ℓ 2 loss/MSE [3].
More recent approaches have used generative adversarial networks(GANs) to generate colorful vivid images, e.g., [52].
One common problem with models trained tocolorize images is that of color bleeding, see Figure 3 for an example of color bleeding producedby the ChromaGAN framework [52].
Our aim is to show that the hybrid approach can smooth localneighborhoods and mitigate color bleeding, at least in some cases.
Model Details To evaluate the hybrid approach, we used the ChromaGAN generative adversarialnetwork for image colorization [52].
The ChromaGAN model consists of a pair of NNs: one is agenerator network, which is responsible for generating the 2 missing color channels, the other is adiscriminator network, which is used to judge how authentic the generated color images are.
Weadded a grid-structured CRF whose parameters are selected via two input networks; a discriminatornetwork that, in this case, is selected to be the generator network in ChrmoaGAN and a parameternet that takes the single channel image as input and produces potential parameters for the CRF.
Theoutput of the CRF inference, combined with input lightness channel, is sent to the discriminatorduring training.
We report only the performance of LSM in this case as SG is a special case that onlyperformed worse in the previous experiments.
Dataset ILSVRC2012 [44] contains roughly 1.3 million images categorized into 1K classes.
Thetraining/hyperparameter tuning follows that of Vitoria et al.
[52], over epochs (10).
Results We evaluated the models using peak signal to noise ratio (PSNR) and structural similarityindex measure (SSIM).
The SSIM value was averaged across RGB channels when calculated againstground truth color images.
Following Vitoria et al.
[52], we evaluated the models on the first 1000images from the ILSVRC2012 validation dataset, ChromaGAN has a 23.09 PSNR / 0.915 SSIM while
Chroma-GAN (LSM) has a 24.19 PSNR / 0.927 SSIM (larger is better in both cases).
Qualitatively,we found that the hybrid model can sometimes significantly reduce color bleeding.
Figure 3 givesthree examples of some visual improvements that were obtained by the hybrid model, which tends tohave less color bleeding around object boundaries, e.g., the orange spot near the dog’s tail disappearsafter adding the CRF.
Additional visual comparisons can be found in the supplementary material.
6.3 Semantic Segmentation
Finally, we consider the semantic segmentation problem, an image classification problem in whicheach pixel in the input image is assigned an individual category, e.g., road, tree, sky.
The input is asingle image, and the output is the per pixel classification.
Model Details We used here is used the DeepLabv3+ [10] architecture to evaluate the hybridapproach.
DeepLabv3+ is an encoder-decoder architecture, with an encoder network as a backboneto extract high level features and a decoder network to do up-sampling.
The input image has size of
H × W × 3 , and the output logit tensor will be of size of H × W × C , where C is the number ofpossible classes.
Thus, for each pixel, there is a logit vector of size C to represent the scores of thatpixel belonging to the corresponding class.
Given that each class score of neighbouring pixels shouldhave dependency on each other, the previous DNN+CRF model that encourages similarities betweenneighbors should also make sense here.
The only difference that is that is we apply a CRF for eachclass but enforce that those CRFs all share the same parameters.
Dataset The COCO-Stuff dataet [7], which contains all images from the COCO-2017 dataset(118,287 training + 5K validation) with pixel-wise annotations for 91 stuff classes, e.g., sea, sky,river, etc., and 1 thing class for objects such as people, car, etc.
COCO-Stuff does not distinguishbetween things.
For training, H = 512 , W = 512 , C = 93 (91 stuff classes, 1 class unlabeled, 1class for things).
We use the Adam optimizer with learning rate 0.0001, both models were trained onthis data set for 30 epochs with batch size 12.
Results We evaluate the learned models using Mean Intersection-Over-Union (MIoU) and Pixel
Accuracy.
Table 2 displays the quantitative results: the LSM CRF yields a mild improvement withrespect to both metrics.
Figure 4 shows the qualitative results of this experiment: Adding the CRFsmooths the segmentation map, eliminating some obvious prediction errors in the pure DNN modelbut over-smoothing in some cases.
Although the CRF cannot correct more severe issues, e.g. distinctpavement and road, it does still produce a smoother overall result.
Discussion
While current trends seem to preference pure DNN solutions for challenging applications in computer vision and other domains, we have demonstrated that CRFs still provide a simple tool tosmooth/regularize and encourage local consistency in DNN outputs.
In addition, we showed thatattractive smoothing via continuous log-supermodular models can yield significant performanceimprovements in practice.
All of this is tied together in an end-to-end framework that scales usingmodern GPUs and approximate variational inference.
The flexible framework evaluated here can easily be incorporated into existing DNN approaches, andperformance improvements in practice range from mild to significant with little additional overheadas part of the training or inference procedures on modern GPUs.
Our hope is that these resultsprovide enough empirical evidence to suggest that hybrid CRF+DNN frameworks be consideredwhen designing new frameworks/architectures for which attractive smoothing is applicable.
Theexperimental approach herein uses limited types of potential functions and limited types of smoothingfor simplicity.
It would be interesting to consider more expressive log-supermodular potentialfunctions (including those of arity larger than two) in future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]