S TRUCTURED E NERGY N ETWORK AS A DYNAMIC LOSS FUNCTION A CASE STUDY WITH MULTI - LABEL C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose SEAL which utilizes an energy network as a trainable loss function for a simple feedfoward network.
Structured prediction energy networks (SPENs) (Belanger & McCallum, 2016; Gygli et al., 2017) have shown that a neural network (i.e. energy network) can learn a reasonable energy function over the candidate structured outputs.
We ﬁnd that rather than using SPENs as a prediction network, using it as a trainable loss function is not only computationally more efﬁcient but also better performing.
As the energy function is trainable, we propose SEAL to be dynamic so it can adapt the energy function to focus on the region where feed-forward models will be affected the most at the time point of model update.
We ﬁnd this to be effective in an ablation study comparing SEAL to the static version (§5) where the energy function is ﬁxed after pretraining.
We show the relation to previous work on the joint optimization model of energy network and feedforward model (INFNET) as we show that it is equivalent to SEAL using margin-based loss if INFNET relaxes their loss function.
Based on the unique architecture of SEAL , we further propose a variant of SEAL that utilizes noise contrastive ranking (NCE) loss that by itself does not perform well as a structured energy network, but embodied in SEAL , it shows the greatest performance among the variants we study.
We demonstrate the effectiveness of SEAL on 7 feature-based and 3 text-based multi-label classiﬁcation datasets.
The best version of SEAL that uses NCE ranking method ( SEAL -NCE) achieves close to 2.85, 2.23 respective F point gain on average over cross-entropy and INFNET for feature-based datasets, excluding one dataset that has an excessively large gain.
Lastly, examining whether the proposed framework is effective on a large pre-trained model as well, we observe SEAL -NCE achieving 0.87, 0.68 respective F1 point gain in average over cross-entropy and INFNET with BERT-based adapter model on text datasets.
I NTRODUCTION
Structured prediction is a popular machine learning task wherein the model learns a mapping function from an input x to a multivariate-structured output y .
Popular examples of this include image segmentation (M¨uller, 2014), extracting parse trees or semantic role labels from a text (Palmer et al., 2010), and multi-label classiﬁcation (Belanger & McCallum, 2016; Gygli et al., 2017).
In structured prediction, the output space Y is often extremely large.
For example, in multi-label classiﬁcation, the size of Y is 2 L where model needs to predict output y ∈ { 0 , 1 } L .
There are two key aspects that a model for structured prediction needs to balance: statistical efﬁciency and computational efﬁciency.
Most models for structured prediction can be categorized into two categories: the feed-forward approach, wherein one learns a neural network that models the direct mapping between the input and the structured output, and the structured approach, in which the model explicitly models the interactions (structure) in the output space.
The feed-forward approach learns all dimensions of y jointly in a conditionally independent manner given input x , relying on the representational power of the network to capture dependencies in output structure implicitly.
While the feed-forward approach is computationally efﬁcient, since it does not capture relationships in the label space, it lacks statistical efﬁciency.
The traditional structured approaches, on the other hand, model a joint probability distribution P ( x , y ) that can capture the label relationships.
However, due to the intractability of modeling the full joint distribution which captures every possible interaction of output space, these approaches resort to limiting the interaction terms to local subsets of the output space (Lafferty et al., 2001; Ghamrawi & McCallum, 2005).
In order to make inference for structured models more efﬁcient, a recent line of work replaces the joint probability with a structured energy function E ( x , y ) (LeCun et al., 2006), which can be thought of as an unnormalized probability distribution, allowing the model to learn arbitrary global dependencies in the output space.
There have been several works that propose efﬁcient approximate inference procedures for structured energy networks using gradient based inference (Belanger et al., 2017; Gygli et al., 2017; Rooshenas et al., 2019), hence also called ’prediction networks’.
These models showed noticeable gains in predictive performance over feed-forward models and graphical models that assume partial structure such as full pairwise potential (Chen et al., 2015; Schwing & Urtasun, 2015).
Despite these efforts, the inference for energy-based models still remains relatively inefﬁcient when compared to the feed-forward approach (Tu & Gimpel, 2019).
Moreover, in our experience, the models using gradient based inference (GBI) are ﬁnicky to train as the training also utilizes GBI, requiring numerous hyperparameters: step size, number of iterations for GBI, and initial point to begin GBI.
This raises a question: Can the energy network be used in a way that is as expressive as a full joint probability, as efﬁcient at inference as a feed-forward approach, and also stable and easy to train?
We believe that using structured energy networks as a parameterized dynamic loss function for feed-forward networks, instead of a prediction network, can fulﬁll all these requirements.
In this paper, we propose the S tructured E nergy
A s L oss ( SEAL ) framework that uses a trainable structured energy network (SEN) 1 as a loss function guiding the training of a feed-forward network.
The key idea is to provide the feed-forward network access to rich relationships in the output space through a learned loss function. We also propose to learn SEN in a dynamic fashion by adjusting the energy function to be conﬁdent with the most up-to-date outputs of the feed-forward network.
We show that learning the loss function dynamically leads to more efﬁcient, more stable, and better performance.
SEAL can be viewed as general-purpose framework where one can plugin various loss functions and architectures to train SEN as well as the feed-forward network.
Through experiments (§4), we analyze the effect of applying different energy losses (e.g. margin-based, regression-based) within SEAL framework.
We also propose the noise-contrastive ranking loss (NCEranking) for SEN which performs the best within the SEAL framework.
To summarize, we introduce a general framework SEAL that interprets structured energy networks (SEN) as a dynamic loss functions.
Through empirical evaluation on the task of multi-label classiﬁcation, we analyse the impact of various loss functions for updating SEN.
Finally, we propose an NCE ranking loss that is uniquely suited for the SEAL framework, and demonstrate its superior performance on 7 feature-based as well as 3 text-based multi-label classiﬁcation datasets, when compared to simple feed-forward approach and various energy based models. Require: ( x , y ) : Training Instance
Require: F Φ : Feedforward Network Require: sampling : True/False ﬂag
Require: optimizer Θ , optimizer Φ Require: T : No. of steps t ← 0 Θ 0 , Φ 0 ← Random initialization while t < T do if sampling then S ← { y ( i ) , i = 1 , . . . , K
| y ( i ) ∼ F Φ t ( x ) } else S ← { F Φ t ( x ) } (cid:46) singleton set t ← t + Update
[Table]  S TRUCTURED E NERGY NETWORK A S L OSS ( SEAL ) This section ﬁrst describes the proposed SEAL framework.
Let X denote the input space, Y = { 0 , 1 } L the output space, and ˜ Y
=
[0 , 1] L the continuous relaxation of Y .
Then the structured energy E Θ : X × ˜ Y → R is deﬁned to be a parameterized function of the input and the continuous relaxation of the output.
The feedforward network F Φ : X → ˜ Y is deﬁned to be a neural network that maps an input to the continuous relaxation of the output space.
We denote j -th training instance pair as ( x ( j ) , y ( j ) )
∈ D and y i ∈ { 0 , 1 } , ˜ y i ∈ [0 , 1] to denote i -th label dimension where i ∈ { 1 , . . .
, L } .
As shown in Figure 1, SEAL consists two loss functions: the energy loss L E that trains structured energy network (Θ) , and feedforward loss L F that guides the training of the feedforward network (Φ) .
We ﬁrst discuss how structured energy network implicitly affects training of the feedforward network by deﬁning L F ( Φ ).
Given a training instance ( x , y ) , the feedforward loss is deﬁned as:
[Equation]
Here, the ﬁrst term involving the energy captures interaction across label space whereas binary cross-entropy considers the predictions of each label independently (More details in Appendix B).
Since the training of feedforward network depends on the quality of the parameterized energy network, it is critical to ﬁnd the parameters Θ that produce the best loss surface for training the feedforward network.
This can be done in two ways: by ﬁrst training the energy separately, or by training the energy and the feedforward network simultaneously.
SEAL uses the latter, which is shown to perform better (see §5) than the former. We denote the former method as SEAL -static for distinction .
In SEAL -static, we ﬁrst estimate the energy network parameter (cid:98) Θ over training data and optimize Φ by plugging ﬁxed (cid:98) Θ into L F .
Given ( x , y ) ∈ D , Φ is trained as equation 2.
[Equation]
We ﬁnally present SEAL which learns (cid:98) Θ t dynamically at step t .
To do so, we alternate the optimization steps of Θ and Φ .
Given a training instance ( x , y ) ∈ B t , a complete training step is given as:
[Equation]
Note that, from (2) to (3), arbitrary ˜ y got replaced with F Φ t − 1 ( x ) in estimating Θ t so that estimation of energy surface depends on model output F Φ t − 1 ( x ) .
Further, the update step for feedforward network Φ relies on Θ t rather than static (cid:98) Θ .
In both SEAL and SEAL -static, test-time predictions are performed solely using the feedfoward network as ˆ y i =
( F Φ ( x ) i ≥ 0 . 5) .
The motivation for proposing dynamic loss function in SEAL is as following.
It would be useful to learn perfect energy surface, if possible, by estimating (cid:98) Θ that is static.
However, doing so is challenging as we are estimating energy surface on the joint space of x , ˜ y , a high-dimensional continuous space with limited data and resource.
Instead, we hypothesize that it is more important to concentrate resource in depicting accurate energy surface around the current input, output pair ( x , F Φ ( x )) .
In SEAL , utilizing the fact that we have access to trainable loss function, we dynamically adapt energy surface so that the region of interest for training feedforward network is well represented.
We now discuss various energy loss (margin-based, regression-based, and contrastive-sampling-based) that can be plugged into L E in SEAL .
For brevity, we use L E (Θ) and L E ( x , y , ˜ y ; Θ) interchangeably where again ( x , y ) denotes labeled data and ˜ y denotes probability vector.
We defer the description of speciﬁc energy network structure to the experiment section (§4) as SEAL framework can work with arbitrary network structures.
Margin-based (L E − margin )
SPEN (Belanger & McCallum, 2016) learned structured energy network with SSVM loss (Taskar et al., 2004;
Tsochantaridis et al., 2004)
so that energy functions learn to have sufﬁcient energy difference, larger than the margin ∆ (˜ y , y ) , between arbitrary output ˜ y and true output y .
To examine the effect of the margin-based loss in SEAL , we follow SPEN and utilize the SSVM loss as
[Equation]
Regression-based (L E − regression )
Deep Value Network (DVN) (Gygli et al., 2017) attempts to learn an energy network which directly outputs a score that is similar to the metric s (˜ y , y ) of interest, such as F1 score, that compares arbitrary ˜
y with true output y .
Following DVN, by making score s ( · )and − E ( · ) to be between [0 , 1] , we express regression loss as cross entropy
[Equation]
Noise-contrastive ranking (L E − NCEranking and L E − ranking )
In SEAL , we are interested in capturing the output region that feedforward network has a high probability.
We ask, rather than just taking a single point ˜
y
= F Φ ( x ) from a feedforward network, whether sampling many discrete binary vectors from ˜ y and taking all those samples into consideration for learning E Θ could lead in better estimation of energy surface.
Motivated from noise contrastive estimation (NCE)
(Ma & Collins, 2018), we ask whether energy network E Θ trained to contrast the true output y from groups of K negative samples drawn from feedforward output F Φ ( x ) can induce a good loss for teaching Φ .
The intuition is as the ˜ y
= F Φ ( x ) becomes better and better, the loss surface teaching feedforward network should become more ﬁne grained as ˜ y and samples from ˜ y will be already close to the true y .
Before we discuss L E − NCEranking of our choice, we ﬁrst review original form of NCE ranking loss from Ma & Collins (2018).
For K samples y ( k ) ∼ P N , k = 1 , . . .
, K drawn from noise distribution P N and rewriting y as y (0) without loss of generality, NCE ranking loss is deﬁned as log
[Equation]
, where s ( x , y
[Equation]
[Equation]
Minimization of this loss makes P ( y | x ; Θ
[Equation]
) as an unbiased estimator of true distribution P ( Y | X ) .
Thus, given perfect energy E ( · ) , minimizing − E Θ ( x , y ) with respect to y in L F would be equivalent to maximizing the estimate of − log P
( y | x ) .
From the presented NCE ranking method, we propose to use P ( y | x ; Φ t ) =
(cid:81) i P ( y i | x ; Φ t ) in place of P N ( y ) .
The novelty of this proposal is that we view the output of feedforward network that we train as a noise distribution, and that the energy model that contrasts noise distribution teaches the noise distribution as well through our SEAL framework (equation 3).
In short, we get L E − NCEranking when we plug in s ( x , y ; Θ) = − E Θ ( x , y ) − log P Φ ( y | x ; Φ t ) into equation 7 , i.e . L E − NCEranking = log
[Equation]
This achieves two beneﬁts that we have in mind in using NCE.
First beneﬁt is that it captures the region of interest by sampling from the output distribution of Φ t .
Second beneﬁt is that it brings more efﬁcient NCE method.
NCE method is known to work best when the noise distribution is close to the data distribution (Gutmann & Hirayama, 2012) but not exactly the same.
In our SEAL learning procedure, we hypothesize that P ( y | x ; Φ t ) becomes closer to the data distribution as training steps proceed.
Lastly, motivated that NCE can estimate the true distribution, we propose yet another loss function that can estimate the difference between the true probability P ( y | x ) and feedforward network probability.
We show in appendix A, if we use plain ranking loss that sets s ( x , y ; Θ) = − E Θ ( x , y ) in equation 7, unbiased estimator of P ( y | x ) now becomes [Equation] x
[Equation]
This means if we take a derivative of E Θ with respect to y , then we are taking derivative with respect to − log P ( y | x ) + log P N ( y ) .
With the trick of replacing the P N ( y ) with P ( y | x ; Φ t ) in SEAL , we hypothesize that loss captured by energy network can focus only on the difference between true and feedforward probability which may end up being more informative surface.
In short, we get plain ranking loss L E − ranking by plugging s ( x , y ; Θ) =
− E Θ ( x , y ) into equation 7 , resulting in
[Equation]
R ELATED W ORK
Learning with dynamic loss: Wu et al.
and
Huang et al.
(2019) attempt to learn a dynamic loss deﬁned by neural network that is tailored to the task-speciﬁc metric m , such as BLEU or 0accuracy. Both approaches try to to learn a loss function that can directly increase model’s metric score on the validation set.
Since m is usually non-differentiable, Huang et al.
(2019) view this problem as a reinforcement learning problem and attempts to learn a loss function that maximizes the expected reward: higher m score in validation set.
On the other hand, Wu et al.
(2018) relax the m to be differentiable and view the feed-forward model update and loss-model update as a connected chain of computation graph, so that loss function parameter ( Θ ) gets updated in relation to feed-forward model ( Φ ) improving the relaxed metric score.
There are three major differences between SEAL and the previous work.
First, energy surface of SEAL is not limited to a speciﬁc parametric form, σ ( y T Θ log F Φ ( x )) 3 , as previous papers do.
Instead, SEAL simply utilizes a neural network to express the loss function as − E Θ ( · ) which allows it to represent a much larger function class.
Second, previous works require manual design of state vector. For example, the state vector used by Wu et al.
(2018) consists of iteration number, validation accuracy, and current loss-function parameter Θ . In contrast, SEAL only requires the input and output pair ( x , F Φ ( x )) of the feed-forward network.
Lastly, since these models were designed for multi-class classiﬁcation, it is non-trivial to extend them to the multi-label classiﬁcation or to the general structured prediction task wherein large number of labels interact.
Structured Prediction Energy Networks: Structured prediction energy networks (Belanger &
McCallum, 2016) and its variants (Gygli et al., 2017; Rooshenas et al., 2019) (referred to as SPENs from here on) learn an energy network E Θ : X × ˜ Y → R and predict the output using gradient-based inference (GBI) with the objective
y = arg
min y E ( x , y ) .
To train energy network, SPENs apply GBI adversarially in order to ﬁnd x , y pair that violates the minimality of the energy the most.
We found this training procedure to be a bit unstable as its success depends on the initial point of GBI.
We discuss the loss function for SPEN (Belanger & McCallum, 2016) and DVN (Gygli et al., 2017) in §2 and use these models as baselines in this work.
Jointly learning Θ and Φ : The Inference network (INFNET) proposed in Tu & Gimpel (2018) is the ﬁrst work that tried to learn structured energy network and feedforward network jointly.
As the name suggests, INFNET proposes a model that mimics the behaviour of SPEN by replacing the expensive gradient-based inference with feedforward networks.
However, just like SPENs, INFNET also resorts to ﬁnding adversarial point that has the maximum energy violation, resulting in an adversarial framework similar to GAN (Goodfellow et al., 2014), where INFNET maximizes, and energy network minimizes the same margin based loss given in equation 5.
INFNET has to serve as the adversarial sampler for training energy network.
While the end goal of INFNET is an efﬁcient inference for ﬁnding output with lowest energy, Having to serve two roles, INFNET suffers from train-, test-time objective mismatch and Tu et al.
(2020) resolves this by augmenting INFNET structure; Two separate feed-forward networks are used, one for prediction and the other as the adversarial sampler.
However, INFNET’s reliance on margin-based losses (marign-rescaled higne and perceptron) still leaves the approach to be adversarial, i.e. min Θ max Φ L E ( x , y , F Φ ( x ); Θ) .
Learning label embeddings for multi-label classiﬁcation Utilizing label representation has been a popular approach for multi-label classiﬁcation (Wang et al., 2018; Xiao et al., 2019; Zhang et al., [Table] These works embed labels as vectors, allowing for richer representations, and then utilizing cross-entropy loss which models individual label probability independently. While the energy network in SEAL represents an output probability, its main purpose is to allow the gradient for each label to depend on other label probabilities, a feature which is absent from these prior works, for which label interactions must be latently encoded in the embedding space.
Eenrgy-based models used for reranking Recently, energy-based models (EBM) have been applied to text generation problems to ﬁrst generate and rerank the generated sentences by EBM score (Deng et al., 2020; Bhattacharyya et al., 2021).
The residual energy modeling of Deng et al.
(2020) shares some similarity to the equation 9,apart from the minor difference that they utilize frozen language model for P N whereas SEAL has P N that constantly learns from the energy models. The major difference is that these previous works have simply used the scores of the energy network, whereas SEAL views the energy network as a loss function, leveraging the gradient from the energy network to further train another neural network.
E XPERIMENTS
In our experiments, we use 10 multi-label classiﬁcation datasets (shown in Table 1) covering varied label space, size, as well as input characteristics.
The ﬁrst 7 are small and medium sized feature based datasets, while the last three Blurb Genre Collection (BGC), RCV (Reuters 1), and New York Times (NYT) are large datasets with raw text as the input.
Energy Network: We use the same structure for the energy network E Θ ( x, y ) as described in Belanger & Mccallum (2016), wherein the energy is the sum of local E local Θ ( x, y ) and global E global Θ ( x, y ) energies deﬁned as:
[Equation]
where b i , v , M , T E contain learnable parameters, σ ( z )
= log(1 + e z ) is the softplus activation function, and T E is a feature network for structured energy network.
Note that due to the presence of E global Θ term in L F , the gradient for feedforward network, ∂ L F y i , can capture the dependency of y i to all L dimensions of y .
The expression for this is provided in Appendix B.
Feature Network: We use the same feature network structure for both the energy and feedforward network denoted as T E , T F respectively.
For the the feature-based datasets, the feature network
T : R d → R h consists of multi-layer perceptron with with softplus nonlinearities.
For the raw text based datasets, we use pre-trained BERT (Devlin et al., 2019) with adapter (Houlsby et al., 2019;
Pfeiffer et al., 2020) as the feature network.
Feedforward Network:
Given a feature network T F that generates features in R h , the feedforward network F Φ ( x ) =
G T F ( x ) , where G ∈ R L × h is a matrix one each [Table]
Training: The models for feature-based datasets are trained on a single TitanX GPU with 14GB CPU memory and the models for BCG, RCV and NYT are trained on m40 GPU.
We use separte ADAM optimizer (Kingma & Ba, 2014) for energy network ( Θ ) and for feedforward network ( Φ ) which optimizes parameters in alternating fashion 5 .
For each minimization loop, we take n E , n F gradient steps respectively for estimating Θ t and Φ t .
We leave these n E , n F as hyperparameters to be tuned.
The best hyper-parameters for each model are found using Bayesian search 6 and reported in Appendix
D. We also report train, inference time and parameter size of different methods in AppendixE.
4. R ESULTS FOR FEATURE BASED DATASETS
The experiment results across 7 feature-based dataset are presented in Table 2.
The table consists of feedforward network only trained with cross-entropy (row 1), energy networks (row 2-4) described in previous section that is evaluated with GBI, and lastly feedforward network trained with SEAL framework with different types losses (row 5-9).
SEAL outperforms cross-entropy loss (CE) on almost all datasets with average gains over CE ranging from +0.6 ( SEAL -L E − margin ) to +2.85 ( SEAL -L E − NCEranking ) F1 points.
We excluded excessive gain of genbase in computing the average as we believe it is an outlier.
Further analysis on the performance on genbase is provided in Appendix F.
Between L E types, L E − NCEranking and L E − ranking performs the best while L E − margin and L E − regression performs similarly on average.
L
E − NCEranking and L E − ranking are the only ones that always outperforms
CE in all 7 datasets whereas the others sometimes perform slightly worse than CE.
With the high gains of L E − NCEranking and L E − ranking , we are likely to conclude using samples rather than a single point can capture the output surface of the feedforward network better.
To examine this hypothesis in more detail, we test sampling approaches on regression-based loss as well.
We deﬁne L E − regression-s = (cid:80)
˜
y ∈ S L E − regression ( x , y ∗ , ˜ y ; Θ) given sample set S .
We take two approaches in collecting these samples: 1.
discrete binary vectors drawn from probability vector
˜
y as done in L E − NCEranking and 2. continuous perturbation of ˜ y with gaussian noise. We found that discrete samples did not make any notable changes between L E − regression and L E − regression-s , however, found that using continuous samples made +0.4 F1 improvements in average and report L E − regression-s as the version which uses continuous sample. We conlcude that using samples are helpful in capturing better energy surface for traininig feedforward network, however, effect and characteristics of sample
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
might differ per types of L E . We believe the sampling is more effective on L E − NCEranking , L E − ranking as groups of samples contribute in relative manner in learning surface in contrast to L E − regression . To further study effect of different L E , we examine whether there is a correlation in performance between the isolated structured energy network and SEAL which utilizes idential L E .
For training isolated structured energy network with margin-based and regression-based energy loss, we follow training approaches of SPEN (Belanger & McCallum, 2016) and DVN (Gygli et al., 2017).
For
NCE, we need access to noise distribution P N .
we set pretrained feedforward network probability
P ( Y | X ; Φ) as P N ( Y ) .
We do not see much of correlation between the two, in other words, characteristics of L E that trains best structured energy network and L E that trains best feedforward network are different.
While NCE seems to perform worst when utilizing L E − NCEranking in isolated manner, the constrastive characteristic of L E − NCEranking and L E − ranking that is tailored to feedforward network seems to be helpful.
Regardless of L E types, we also obesreve that it is much more stable and usually higher-performing to train with SEAL than to train energy network separately.
In Appendix C, we also show compare individual energy network as a prediction model compared to SEAL in detail.
4. R ESULTS FOR LARGE TEXT DATASETS
[Table]
In order to test the effectiveness of SEAL in the scope of large pre-trained models, we examine text datasets with pre-trained BERT (Devlin et al., 2019) with adapter (Houlsby et al., 2019; Pfeiffer et al., 2020) as the feature network T E and T F as shown in Table 3.
Considering the computational expense of running hyper-parameter searches on BERT, based on Table 2, we choose the best-performing SEAL (NCEranking) to compare with baselines: SEAL -margin and cross-entropy.
The NYT subset we used has almost 17x larger training data than the largest feature-based data and SEAL framework outperforms cross-entropy ﬁne-tuning.
Overall, the Table 3 shows that SEAL , especially SEALNCEranking, is effective on pre-trained models that utilizes large text datasets as well.
On average, SEAL -NCEranking gains 0.87, 0.68 F1 points over cross entropy and SEAL -margin, respectively, on text datasets as shown in Table 3.
F URTHER ANALYSIS
SEAL without cross entropy: Figure 2 shows the performance of SEAL when the cross-entropy loss (CE) is completely removed from equation 1. We ﬁrst observe that surprisingly that SEAL itself [Figure]
NCEraking [Figure] without resorting to CE learns a model that sometimes outperforms model trained with CE ( cal and spo fun).
However, we also observe that SEAL without CE is unstable as performance varies a lot across different datasets.
This tells us that SEAL requires a signal that can bring the model F Φ ( · ) into reasonable region ﬁrst in order to demonstrate stable performance shown in Table 2.
Another interesting observation is that SEAL -NCEranking model when it is often the weakest without CE can give the largest improvement when combined with CE together in SEAL as shown in Table 2.
In short, we ﬁnd CE to bring a stable synergy with energy network loss within SEAL : surpassing each individual’s performance signiﬁcantly when combined together.
While there could be multiple techniques to stabilize learning of SEAL besides adding CE loss, such as pre-training techniques, we leave those interesting studies as future work.
Effect of dynamic loss computation: We now compare the effect of dyanmic loss learning by comparing SEAL to SEAL -static in Figure 3We observe that pretrained energy networks with SEALstatic also results in competitve results, often better performing than the simple cross-entropy loss.
However, we oberve that in average, the dynamic loss learnig of SEAL outperforms SEAL -static for L E − regression , and L E − NCEranking .
Effect of applying ranking loss directly on F Φ
With the large improvement that L E − NCEranking and L E − ranking has, we examine whether the gain is actually coming from SEAL framework or simply from the power of ranking loss itself which can be applied to feedforward network F Φ directly.
We conduct this ablation study over relatively small datasets: genbase, cal500 and delicious.
While we observed a near +10 F1 point in genbase, we saw decrease in performance in cal500 (-1.0 F1) and delicious (-2.3 F1) compared to plain cross-entropy model. With these experiments, we notice two trends.
First, we again observe that genbase is an outlier where capturing structure can greatly increase the performance although not as much as SEAL framework helped. Two, we conﬁrm that the ranking loss without energy network is not very effective in training feedforward model F Φ .
This again conﬁrms that the energy network E Θ plays a major role to train F Φ in the SEAL framework.
C ONCLUSION
We propose SEAL : a framework that can adopt structured energy network as a trainable loss function for training feedforward network.
Through examining different energy losses that trains energy network, we show that SEAL is a general framework which is effective over different loss functions and different architectures, such as MLP and BERT-based adapter.
Through extensive result onfeature-based and 3 text-based datasets, we show that SEAL brings synergy between isolated energy newtork and cross entropy function: performing better and in more stable manner when combined together.
Lastly, through ablation study, we examine the beneﬁt of dyanmic loss learing which in average brings a gain over static version of SEAL , i.e. SEAL -static.
This research opens up doors for various future work, including but not limited to, application of learned energy loss in unsupervised dataset and exploring different architectures of energy network.
In the long horizon, the authors are interested in whether general pretrained scoring neural networks, such as BLEURT (Sellam et al., 2020) and BERT-score (Zhang et al., 2019), could serve as a loss function when it can provide a backpropagatable gradients to the output space.
A PPENDIX A NCE RANKING METHOD WITH s ( x , y ; Θ) = − E Θ ( x , y )
As pointed out by Ma & Collins (2018) as well, If we set s ( x , y ; Θ) = − E Θ ( x , y ) , we can regard this this learning with E (cid:48)
Θ ( x , y ) =
E Θ ( x , y ) + log P N
( y ) .
As pointed out earlier, this leads the model to estimate P ( y | x with
[Equation]
Plugging in E Θ (cid:48) as E Θ ( x , y ) + log P N ( y ) once again, we can see that P ( Y | X ) is estimated by
[Equation]
B G RADIENT TO THE LABEL REPRESENTATIONS
As described in section 4, the energy component of loss L F , for the feedforward network, is the sum of following two parts:
[Equation]
and
[Equation]
The gradient of E global w.r.t y is given by the following vector of length
L
[Equation]
While the gradient of cross entropy l ( y, y ∗ ) w.r.t.
y is given by the following vector of length
L
[Equation]
(14) where ⊗ is element-wise product.
Note the dependency of the gradient for a particular component y i of y on other component in the expressions above.
The gradient of cross entropy ( ∂l∂y i ) only depends on its label dimension i : y i and y ∗ i .
In contrast, in the case of the energy function as loss (SEAL), the gradient ∂ L F ∂y i depends on all L dimensions of y due to the product term My in ∂E global ∂y i .
C C OMPARISON OF SEAL AND SEPARATE ENERGY NETWORK
In the following ﬁgures, we compare the F1 scores of SEAL framework with L E − regression (Figure 4),
L E − NCEranking (Figure 5) and L E − margin (Figure 6) on validation set with its energy-only counterpart.
We further report validation F1 scores for the output of GBI starting from the samples drawn from ˆ y and randomly.
We show this comparison to evaluate the performance of the energy network on points around ˆ y and random points.
[Figure]
[Figure] [Figure]
D H YPERPARAMETERS
The following tables tabulate the hyper-parameters for SEAL framework for each dataset.
We further ﬁx the number of layers, layer-dimensions and dropout in feed-forward network based on the conﬁguration of the best model trained using cross-entropy.
We ﬁx the λ 2 = 1 in equation 1 to reduce degree of freedom in hyperparameter search.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
E A NALYSIS ON PARAMETER SIZE AND SPEED OF INFERENCE AND TRAINING
Here, we present the analysis on number of parameters (Table 11), inference speed (Table 12))
[Table]
and training speed (Table 13) that different methods utilize per dataset.
In summary, SEAL requires approximately twice number of parameters compared to cross-entropy loss on training time, however it requires the same amount of parameter on inference time.
Similarly, train time (sec) per epoch of SEAL is slower than that of cross-entropy and energy network as
SEAL requires multiple backpropagation steps for both energy network and feedforward network.
Nonetheless, again, at inference time
[Table]
the speed of CE and SEAL is the same and is much faster (2x-7x) than the inference of energy network.
The analysis we present below are on the seven feature-based dataset, however the trend is similar for text-based dataset as well: twice parameter size on training time but equal parameter and runtime in inference time with CE method.
The only difference in the text-based MLC is, due to running on a much larger BERT-baed models, that we only run one step for each update of feedforward network and energy network.
Thus, unlike Table 13, train time of SEAL is approximately twice that of CE [Table] [Table] [Table]
F D ATA - SPECIFIC ANALYSIS
In this section, we analyze why certain datasets behaves differently. F. GENBASE Here, we analyze the factors behind SEAL achieving almost perfect F1 score on the genbase dataset. It seems genbase stands out with its small label size (27 which is the smallest among Table 1) and with very clear pattern in the label space.
Upon analysis, out of 27 labels, we found that 6 labels only occur as a singleton (by itself), 10 labels only occur as non-singleton, and 7 labels do not participate at all.
This leaves only 4 labels occurring by themselves as well as with others.
Not only that, only 7% (35/500) of training instances have more than one active label. In this peculiar setting, we conjecture feedforward networks with cross-entropy (CE) loss can easily learn singletons, and energy networks would easily learn co-occurrence, but learning both might be confusing for these two models at the opposite end of the spectrum. We believe the synergy of CE and a loss from the energy network enables capturing the best of both worlds and achieving a near-perfect score that none of the approaches by itself achieves. As can seen in Table 2, neither cross-entropy nor energy networks are nearly as strong as SEAL methods.
In Table 2, it seems SEAL -NCEranking and SEAL -Ranking are particularly well performing on cal500 and delicious. It turns out that cal500 and delicious have a very high diversity of 1 and 0. (Reference: MLC data repo).
Diversity of 1 means that each data point holds a unique label set.
It seems that the exposure to multiple samples and evaluations of their relative scores in NCEranking loss function can be helpful in a high-diversity setting.