S TRUCTURED E NERGY N ETWORK AS A DYNAMIC LOSS FUNCTION A CASE STUDY WITH MULTI - LABEL C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose SEAL which utilizes an energy network as a trainable loss function for a simple feedfoward network.
Structured prediction energy networks (SPENs) (Belanger & McCallum, 2016; Gygli et al., 2017) have shown that a neural network (i.e. energy network) can learn a reasonable energy function over the candidate structured outputs.
We ﬁnd that rather than using SPENs as a prediction network, using it as a trainable loss function is not only computationally more efﬁcient but also better performing.
As the energy function is trainable, we propose SEAL to be dynamic so it can adapt the energy function to focus on the region where feed-forward models will be affected at the time point of model update.
We ﬁnd this to be effective in an ablation study comparing SEAL to the static version (§5) where the energy function is ﬁxed after pretraining.
We show the relation to previous work on the joint optimization model of energy network and feedforward model (INFNET) as we show that it is equivalent to SEAL using margin-based loss if INFNET relaxes their loss function.
Based on the unique architecture of SEAL , we further propose a variant of SEAL that utilizes noise contrastive ranking (NCE) loss that by itself does not perform well as a structured energy network, but embodied in SEAL , it shows the greatest performance among the variants we study.
We demonstrate the effectiveness of SEAL on 7 feature-based and 3 text-based multi-label classiﬁcation datasets.
The best version of SEAL that uses NCE ranking method achieves close to +2.85, +2.23 respective F1 point gain on average over cross-entropy and INFNET for feature-based datasets, excluding one outlier that has an excessive gain of +50.0 F1 points.
Lastly, examining whether the proposed framework is effective on a large pre-trained model as well, we observe SEAL achieving +0.87 F1 point gain in average on top of BERT-based adapter model on text datasets.
I NTRODUCTION
Structured prediction is a popular machine learning task wherein the model learns a mapping function from an input x to a multivariate-structured output y .
Popular examples of this include image segmentation (M¨uller, 2014), extracting parse trees or semantic role labels from a text (Palmer et al., 2010), and multi-label classiﬁcation (Belanger & McCallum, 2016; Gygli et al., 2017).
In structured prediction, the output space Y is often extremely large.
For example, in multi-label classiﬁcation, the size of Y is 2 L where model needs to predict output y ∈ { 0 , 1 } L .
There are two key aspect that a model for structured prediction needs to balance: statistical efﬁciency and computational efﬁciency.
Most models for structured prediction can be categorized into two categories: the feed-forward approach wherein one learns a neural network that models the direct mapping between the input and the structured output and the structured approach in which the model explicitly models the interactions (structure) in the output space.
The feed-forward approach learns all dimensions of y jointly in a conditionally independent manner given input x , relying on the representational power of the network to capture the dependencies of output structure implicitly.
While the feed-forward approach is computationally efﬁcient, since it does not capture the relationships in the label space, it lacks statistical efﬁciency.
The traditional structured approaches, on the other hand, model a joint probability distribution P ( x , y ) that can capture the label relationships.
However, due to the intractability of modeling full joint distribution which captures every possible interaction of output space, these approaches resort to limiting the interaction terms to local subsets of the output space (Lafferty et al., 2001; Ghamrawi & McCallum, 2005).
In order to make the inference for structured models more efﬁcient, a recent line of work replaces the joint probability with a structured energy function E ( x , y ) (LeCun et al., 2006), which can be thought of as unnormalized probability distribution, allowing the model to learn arbitrary global dependencies in the output space.
There have been several works that propose efﬁcient approximate inference procedures for structured energy network using gradient based inference (Belanger et al., 2017; Gygli et al., 2017; Rooshenas et al., 2019), hence also called ’prediction network’.
These models showed noticeable gains in predictive performance over feed-forward models and graphical models that assume partial structure such as full pairwise potential (Chen et al., 2015; Schwing & Urtasun, 2015).
Despite these efforts, the inference for energy-based models still remains relatively inefﬁcient when compared to feed-forward approach (Tu & Gimpel, 2019).
Moreover, based on our experience, the models using gradient based inference (GBI) are ﬁnicky to train as training procedure also utilizes
GBI that requires numerous hyperparameters: initial point to begin, step size, and number of iterations for GBI.
This raises a question: Can the energy network be used in a way that is as expressive as a full joint probability, as efﬁcient at inference as a feed-forward approach, and also stable and easy to train?
We believe that using structured energy networks as a parameterized dynamic loss functions for feed-forward networks, instead of a prediction network, can fulﬁl all these requirements.
In this paper, we propose a framework called S tructured E nergy
A s L oss, or SEAL , that uses the trainable structured energy network (SEN) 1 as a loss function guiding the training of a feed-forward network.
The key idea is to provide the feed-forward network access to rich relationships in the output space through a learned loss function, and do this in a dynamic manner by updating the loss based on the current outputs of the feed-forward network.
We show that learning the loss function dynamically, is efﬁcient, stable and produces the best performance.
SEAL can be viewed as general-purpose framework where one can plugin various loss functions and architectures to train SEN as well as the feed-forward network.
Through experiments (§4), we compare the impact of using different losses (e.g. margin-based, regression-based, etc.)
to update the SEN.
We identify the beneﬁts and shortcoming of each and propose the noise-contrastive ranking loss (NCEranking) which works best with the SEAL framework.
To summarize, we introduce a general framework SEAL that interprets structured energy networks (SEN) as a dynamic loss functions.
Through empirical evaluation on the task of multi-label classiﬁcation, we analyse the impact of various loss functions for updating SEN.
Finally, we propose a NCE ranking loss that is uniquely suited for the SEAL framework, and demonstrate its superior performance on 7 feature-based as well as 3 text-based multi-label classiﬁcation datasets, when compared to simple feed-forward approach and various energy based models.
S TRUCTURED E NERGY NETWORK A S L OSS ( SEAL )
This section ﬁrst describes the proposed framework S tructured E nergy network
A s L oss ( SEAL ).
Then, we introduce various energy-network-learning objectives that can be plugged into SEAL .
Let
X denote the input space, Y = { 0 , 1 } L the output space, and ˜ Y
=
[0 , 1] L the continuous relaxation of Y .
Then the structured energy E Θ : X × ˜ Y → R is deﬁned to be a parameterized function of the input and the continuous relaxation of the output.
The feedforward network F Φ : X → ˜ Y is deﬁned to be a neural network that maps an input to the continuous relaxation of the output space.
We denote j -th training instance pair as ( x ( j ) , y ( j ) )
∈ D and y i ∈ { 0 , 1 } , ˜ y i ∈ [0 , 1] to denote i -th label dimension where i ∈ { 1 , . . .
, L } .
As shown in Figure 1, SEAL consists two loss functions: the energy loss L E that trains structured energy network (Θ) , and feedforward loss L F that guides the training of the feedforward network (Φ) .
We ﬁrst discuss how structured energy network implicitly affects training of the feedforward network by deﬁning L F ( Φ ).
Given a training instance ( x , y )
the feedforward loss is deﬁned as:
[Equation]
Here, the ﬁrst term involving the energy captures interaction across label space whereas binary cross-entropy considers the predictions of each label independently (More details in Appendix B).
Since the training of feedforward network depends on the quality of the parameterized energy network, it is critical to ﬁnd the parameters Θ that produce the best loss surface for training the feedforward network.
This can be done in two ways: by ﬁrst training the energy separately, or by training the energy and the feedforward network simultaneously.
SEAL uses the latter, which is shown to perform better (see §5) than the former, which we denote as as SEAL -static for distinction .
In SEAL -static, we ﬁrst estimate the energy network parameter (cid:98) Θ over training data and optimize Φ by plugging (cid:98) Θ into L F .
Given ( x , y ) ∈ D , Φ is trained as equation 2,equation 2.
[Equation]
We ﬁnally present SEAL which learns (cid:98) Θ t dynamically at step t .
To do so, we alternate the optimization steps of L E eq.(2) and L F eq.(2).
Given a training instance ( x , y ) ∈ B t , a complete training step is given as:
[Equation]
Note that, from (2) to (3), arbitrary ˜ y got replaced with F Φ t − 1 ( x ) in estimating Θ t so that estimation of energy surface depends on model output F Φ t − 1 ( x ) .
Further, the update step for feedforward network Φ relies on Θ t rather than static (cid:98) Θ .
In both SEAL and SEAL -static, test-time predictions are performed solely using the feedfoward network as ˆ y i =
( F Φ ( x ) i ≥ 0 . 5) .
The motivation for proposing dynamic loss function in SEAL is as following.
It would be useful to learn perfect energy surface, if possible, by estimating (cid:98) Θ that is static.
However, doing so is challenging as we are estimating energy surface on the joint space of x , ˜ y , a high-dimensional continuous space with limited data and resource.
Instead, we hypothesize that it is more important to concentrate resource in depicting accurate energy surface around the input, output pair ( x , F Φ ( x )) .
In SEAL , utilizing the fact that we have access to trainable loss function, we dynamically adapt energy surface so that the region of interest for training feedforward network is well represented.
We now discuss various energy loss (margin-based, regression-based, and contrastive-sampling-based) that can be plugged into L E in SEAL .
For brevity, we use L E (Θ) and L E ( x , y , ˜ y ; Θ) interchangeably where again ( x , y ) denotes labeled data and ˜ y denotes probability vector.
We defer the description of speciﬁc energy network structure to the experiment section (§4) as SEAL framework can work with arbitrary network structures.
Margin-based (L E − margin )
SPEN (Belanger & McCallum, 2016) learned structured energy network with SSVM loss (Taskar et al., 2004;
Tsochantaridis et al., 2004)
so that energy functions learn to have sufﬁcient energy difference, larger than the margin ∆ (˜ y , y ) , between arbitrary output ˜ y and true output y .
To examine the effect of the margin-based loss in SEAL , we follow SPEN and utilize the SSVM loss as
[Equation]
Regression-based (L E − regression )
Deep Value Network (DVN) (Gygli et al., 2017) attempts to learn an energy network which directly outputs a score that is similar to the metric s ( ˜ y , y ) of interest, such
Algorithm
Require: ( x , y ) :
Training Instance
Require: F Φ : Feedforward Network Require: sampling : True/False ﬂag
Require: optimizer Θ , optimizer Φ Require:
T : No. of steps t ← 0 Θ 0 , Φ 0 ← Random initialization while t < T do if sampling then
S ← { y ( i ) , i = 1 , . . .
, K
| y
( i ) ∼ F Φ t ( x ) } else S ← { F Φ t ( x ) } (cid:46) singleton set t ← t +
Update Θ as Eqn. 3 using ˜ y ∈ S and optimizer Θ Update Φ as Eqn.
4 using optimizer Φ as F1 score, that compares arbitrary ˜ y with true output y . Following DVN, by making score s ( · )and − E ( · ) to be between [0 , 1] , we express regression loss as cross entropy
[Equation]
Noise-contrastive ranking (L E − NCEranking and L E − ranking )
In SEAL , we are interested in capturing the output region that feedforward network has a high probability.
We ask, rather than just taking a single point ˜
y
= F Φ ( x ) from a feedforward network, whether sampling many discrete binary vectors from ˜ y and taking all those samples into consideration for learning E Θ could lead in better estimation of energy surface.
Motivated from noise contrastive estimation (NCE)
(Ma & Collins, 2018), we ask whether energy network E Θ trained to contrast the true output y from groups of K negative samples drawn from feedforward output F Φ ( x ) can induce a good loss for teaching Φ .
The intuition is as the ˜ y
= F Φ ( x ) becomes better and better, the loss surface teaching feedforward network should become more ﬁne grained as ˜ y and samples from ˜ y will be already close to the true y .
Before we discuss L E − ranking of our choice, we ﬁrst review original form of NCE ranking loss from
Ma & Collins (2018).
For K samples y ( k ) ∼ P N , k = 1 , . . .
, K drawn from noise distribution P N and rewriting y as y (0) without loss of generality, NCE ranking loss is deﬁned as
[Equation]
, where s
[Equation]
x , y
[Equation]
Minimization of this loss makes P ( y | x ; Θ
[Equation]
as an unbiased estimator of true distribution P ( Y | X ) .
Thus, minimizing − E Θ ( x , y ) with respect to y in L F would be equivalent to maximizing the estimate of − log P
( y | x ) .
From the presented NCE ranking method, we propose to use P ( y | x ; Φ t ) =
(cid:81) i P ( y i | x ; Φ t ) in place of P N ( y ) .
The novelty of this proposal is that we view the output of feedforward network that we train as a noise distribution, and that the energy model that contrasts noise distribution teaches the noise distribution as well through our SEAL framework (equation 3).
In short, we get L E − NCEranking when we plug in s ( x , y ; Θ) = − E Θ ( x , y ) − log P Φ ( y | x ; Φ t ) into equation 7 , i.e .
[Equation]
k =1 exp (cid:0) s ( x , y ( k ) ; Θ) (cid:1) , where s ( x , y ; Θ) = − E Θ ( x
, y ) − log P Φ ( y | x ; Φ t ) .
This achieves two beneﬁts that we have in mind in using NCE.
First beneﬁt is that it captures the region of interest by sampling from the output distribution of Φ t .
Second beneﬁt is it brings more efﬁcient NCE method.
NCE method is known to work best when the noise distribution is close to the data distribution (Gutmann & Hirayama, 2012) but not exactly the same.
In our SEAL learning procedure, we hypothesize that P ( y | x ; Φ t ) becomes closer to the data distribution and stabilizes to a certain distribution as it starts to converge.
Lastly, motivated that NCE can estimate the true distribution, we propose yet another loss function that can estimate the difference between the true probability P ( y | x ) and feedforward network probability.
We show in appendix A, if we use plain ranking loss that sets s ( x , y ; Θ) = − E Θ ( x , y ) in equation 7, unbiased estimator of P ( y | x ) now becomes
[Equation]
This means if we take a derivative of E Θ with respect to y , then we are taking derivative with respect to − log P ( y | x ) + log P N ( y ) .
With the trick of replacing the P N ( y ) with P ( y | x ) in SEAL , we hypothesize that loss captured by energy network can focus only on the difference between true and feedforward probability which maybe end up being more informative surface.
In short, we get plain ranking loss L E − ranking by plugging s ( x , y ; Θ) =
− E Θ ( x , y ) into equation 7 , which results in equation 9 ,
[Equation]
R ELATED W ORK
Learning with dynamic loss: Wu et al.
and
Huang et al.
(2019) attempts to learn a dynamic loss deﬁned by neural network that is tailored to the metric used for the task.
Both these models try to to learn a loss function that directly increases task-speciﬁc metric , such as BLEU or 0-1 accuracy, by evaluating it on validation set.
Since metric m is usually non-differentiable Huang et al.
(2019) views this problem as a reinforcement learning problem and attempts to learn a loss function that maximizes the expected reward.
On the other hand, Wu et al.
(2018) relaxes the metric function to be differentiable and views the feed-forward model update and loss-model update as a connected chain of computation graph, so that loss function parameter ( Θ ) gets updated in relation to feed-forward model ( Φ ) improving the relaxed metric score.
There are three major differences between these models and SEAL .
First, energy surface of SEAL is not limited to a speciﬁc parametric form, like σ ( y T Θ log F Φ ( x )) for Wu et al.
Instead,
SEAL simply uses a scalar valued neural network to express loss function − E Θ ( · ) which allows it to use a much larger function class.
Second, these models require manual design of state vector, for example the state vector used by Wu et al.
(2018) consists of iteration number, validation accuracy, and current loss-function parameter Θ , whereas SEAL alleviates this requirement by using only input, output pairs ( x , F Φ ( x )) of the feed-forward network.
Lastly, these models were designed for multi-class classiﬁcation, and it is non-trivial to extend them for tasks with extremely large label space like structured prediction.
Structured Prediction Energy Networks: Structured prediction energy networks (Belanger &
McCallum, 2016) and its variants (Gygli et al., 2017; Rooshenas et al., 2019) (referred to as SPENs from here on) learn an energy network E Θ : X × ˜ Y → R and ﬁnds
y = arg
min y E ( x , y ) by applying gradient-based inference (GBI).
To train energy network, SPENs apply GBI adversarially in order to ﬁnd x , y pair that violated the minimality of the energy the most.
We found this training procedure to be a bit unstable as its success depends on the initial point of GBI.
We discuss detailed form of loss function for SPEN (Belanger & McCallum, 2016) and DVN (Gygli et al., 2017) in § and provide these as baselines in this work.
Jointly learning Θ and Φ : The Inference network (INFNET) proposed in Tu & Gimpel (2018) is the ﬁrst work that tried to learn structured energy network and feed-forward network jointly.
As the name suggests, INFNET is focused in bringing efﬁcient inference-time feedforward network that can mimic behaviour of SPEN while bypassing expensive gradient-based inference at test time.
However, just like SPENs, INFNET also resorts to ﬁnding adversarial point that has the maximum energy violation.
This results in adversarial loss similar to GAN (Goodfellow et al., 2014)
where INFNET
[Table]
maximizes and energy network minimizes the SSVM loss. While the end goal of INFNET is in efﬁcient inference for ﬁnding output with lowest energy, it also has to serve as the adversarial sampler for training energy network.
Having to serve two roles, INFNET has train-, test-time objective mismatch as (Tu et al., 2020), which is resolved in Tu et al.
(2020) by using augmented INFNET structure, wherein two separate feed-forward networks are used, one for prediction and the other as the adversarial sampler.
However, INFNET’s reliance on margin-based loss still leaves train-, test-time mismatch.
In contrast, SEAL has completely independent loss functions for updating the energy and feed-forward network.
On feed-forward ( Φ ) update, SEAL simply update the model in the direction that reduces energy deﬁned by structured energy network.
INFNET is a special case of
SEAL (i.e. SEAL with L E − margin ) if the margin is removed from the SSVM loss for the feedforward model udpate.
In fact, Tu et al.
(2020) reports that this special case leads to more efﬁcient and stable learning of feedforward network in their ablation study.
We use this special case of INFNET to represent SEAL with L E − margin in the main experiment.
E XPERIMENTS (R ESULTS )
In our experiments, we use 10 multi-label classiﬁcation datasets (shown in Table 1) covering varied label space, size, as well as input characteristics.
The ﬁrst 7 are small and medium sized feature based datasets, while the last three Blurb Genre Collection (BGC), RCV (Reuters 1), and New York Times (NYT) are large datasets with raw text as the input.
Energy Network: We use the same structure for the energy network E Θ ( x, y ) as described in Belanger & Mccallum (2016), wherein the energy is the sum of local E local Θ ( x, y ) and global E global Θ ( x, y ) energies deﬁned as:
[Equation]
where b i , v , M , T E contain learnable parameters, σ ( z )
= log(1 + e z ) is the softplus activation function, and T E is a feature network for structured energy network.
Note that through E global Θ term, the gradient for feedforward network, ∂ L F y i , can capture the dependency of y i to all L dimensions of y .
Please refer to Appendix B for more details.
Feature Network: We use the same feature network structure for both the energy and feedforward network denoted as T E , T F respectively.
For the the feature-based datasets, the feature network
T : R d → R h consists of multi-layer perceptron with with softmax nonlinearities.
For the raw text based datasets, we use pre-trained BERT (Devlin et al., 2019) with adapter (Houlsby et al., 2019;
Pfeiffer et al., 2020) as the feature network [Table]
Feedforward Network:
Given a feature network T F that generates features in , the feedforward network F Φ ( x ) =
G T F ( x ) , where G ∈ R L × h is a matrix consisting of learnable embeddings, one row of embedding for each label.
Training: The models for feature-based datasets are trained on a single TitanX GPU with 14GB CPU memory and the models for BCG, RCV and NYT are trained on m40 GPU.
We use separte ADAM optimizer (Kingma & Ba, 2014) for energy network ( Θ ) and for feedforward network ( Φ ) which optimizes parameters in alternating fashion 5 .
For each minimization loop, we take n E , n F gradient steps respectively for estimating Θ t and Φ t .
We leave these n E , n F as hyperparameter to be tuned.
The best hyper-parameters for each model are found using Bayesian search 6 and report it in
Appendix D. We also report analysis on train-, inference-time speed and number of parameters used in Appendix E.
4. R ESULTS FOR FEATURE BASED DATASETS
The experiment results across 7 feature-based dataset are presented in Table 2.
The table consists of feedforward network only trained with cross-entropy (row 1), energy networks (row 2-4) described in previous section that is evaluated with GBI, and lastly feedforward network trained within SEAL framework with the dispaly of L E type used (row 5-9).
SEAL outperforms cross-entropy loss (CE) on almost all datasets, average gains over CE ranging from +0.6 ( SEAL -L E − margin ) - +3 ( SEAL -L E − NCEranking ) F1 points in average when the CE results are around 30-50 F1.
We excluded highest gain in computing the average.
The highest gain came form genbase, where the gains are around +50 F1 points (near 100% improvement).
Among various L E s, L E − NCEranking and L E − ranking performs the best when L E − margin and L E − regression performs similary in average.
L
E − NCEranking and L E − ranking are the only ones that always outperforms
CE in all 7 datasets whereas L E − margin and L E − regression sometimes performs slightly worse, but on par, than CE in eurlex and delicious.
With the high gains of L E − NCEranking and L E − ranking , we are likely to conclude using samples rather than a single point can capture the output surface of the feedforward network better.
To examine this hypothesis in more detail, we test sampling approaches on regression-based loss as well.
We deﬁne L E − regression-s = (cid:80)
˜
y ∈ S L E − regression ( x , y ∗ , ˜ y ; Θ) given sample set S .
We take two approaches in collecting these samples: 1.
discrete binary vectors drawn from probability vector
˜
y as done
[Figure]
To further study effect of different L E , we examine whether there is a correlation in performance between the isolate structured energy network and SEAL which utilizes idential L E .
For training isolated structured energy network with margin-based and regression-based energy loss, we follow training approaches of SPEN (Belanger & McCallum, 2016) and DVN (Gygli et al., 2017).
For
NCE, we need access to noise distribution P N .
we set pretrained feedforward network probability
P ( Y | X ; Φ) as P N ( Y ) .
We do not see much of correlation between the two, in other words, characteristics of L E that trains best structured energy network and L E that trains best feedforward network are different.
While NCE seems to perform worst when utilizing L E − NCEranking in isolated manner, the constrastive characteristic of L E − NCEranking and L E − ranking that is tailored to feedforward network seems to be helpful.
Regardless of L E types, we also obesreve that it is much more stable and usually higher-performing to train with SEAL than to train energy network separately.
In Appendix C, we also show compare individual energy network as a prediction model compared to SEAL in detail.
4. R ESULTS FOR LARGE TEXT DATASETS
[Table] In order to test the effectiveness of SEAL in the scope of large pre-trained models, we examine text datasets with pre-trained BERT (Devlin et al., 2019) with adapter (Houlsby et al., 2019; Pfeiffer et al., 2020) as the feature network.
Considering the computational expense of running hyper-parameter searches on BERT, based on Table 2, we choose the best performing SEAL (NCEranking) to compare with baselines: SEAL -margin and cross-entropy.
The NYT subset we used has almost 17x larger training data than the largest feature-based data and SEAL framework outperforms cross-entropy ﬁne-tuning.
This shows that SEAL , especially SEAL -NCEranking, is effective in pretrained models that utilizes large datasets as well.
In average, SEAL -NCEranking gains 0.87, 0.68 F1 points over cross entropy and SEAL -margin in text datset as shown in Table 3.
F URTHER ANALYSIS
SEAL without cross entropy: Figure 2a shows the performance of SEAL when the cross-entropy loss (CE) is completely removed from equation 1.
We ﬁrst observe that surprisingly that SEAL itself without resorting to CE learns a model that sometimes, cal500 and spo fun, outperforms model trained with CE.
However, we also observe that SEAL without CE is unstable as performance varies a lot across different datasets.
This tells us that SEAL requires a signal that can bring the model F Φ ( · ) into reasonable region ﬁrst in order to demonstrate stable performance shown in Table 2.
Another interesting observation is that SEAL -NCEranking model when it is often weakest without CE can give the largest improvement when combined with CE together in SEAL as shown in Table 2.
In short, we ﬁnd CE to bring a stable synergy with energy network loss within SEAL : surpassing each individual’s performance signiﬁcantly when combined together.
While there could be multiple techniques to stabilize learning of SEAL besides adding CE loss, including pretraining techniques, we leave those intersting studies as future work.
Effect of dynamic loss computation: We now compare the effect of dyanmic loss learning by comparing SEAL to SEAL -static in Figure 2b.
We observe that pretrained energy networks with
SEAL -static also results in competitve results, often better performing than the simple cross-entropy loss.
However, we oberve that in average, the dynamic loss learnig of SEAL outperforms SEAL -static for L E − regression , and L E − NCEranking .
Effect of applying ranking loss directly on F Φ
With the large improvement that L E − NCEranking and L E − ranking has, we examine whether the gain is actually coming from SEAL framework or simply from the power of ranking loss itself which can be applied to feedforward network F Φ directly.
We conduct this ablation study over relatively small datasets: genbase, cal500 and delicious.
While we observed a near +10 F1 point in genbase, we saw decrease in performance in cal500 (-1.0 F1) and delicious (-2.3 F1) compared to plain cross-entropy model.
With this we again see that genbase is an outlier where capturing structure really helps, however, we conﬁrm that the ranking loss without energy network is not effective in training feedforward model F Φ .
This again conﬁrms that the energy network E Θ plays a major role in training F Φ .
C ONCLUSION
We propose SEAL : a framework that can adopt structured energy network as a trainable loss function for training feedforward network.
Through examining different energy losses that trains energy network, we show that SEAL is a general framework which is effective over different loss functions and different architectures, such as MLP and BERT-based adapter.
Through extensive result onfeature-based and 3 text-based datasets, we show that SEAL brings synergy between isolated energy newtork and cross entropy function: performing better and in more stable manner when combined together.
Lastly, through ablation study, we examine the beneﬁt of dyanmic loss learing which in average brings a gain over static version of SEAL , i.e. SEAL -static.
This reserach opens up a door for various future work, including but not lmited to, application of learned energy loss in unsupervised dataset and exploring different architectures of energy network.
In the long horizon, the authors are interested in whether general pretrained scoring neural networks, such as BLEURT (Sellam et al., 2020) and BERT-score (Zhang et al., 2019), could serve as a loss function when it provides a backpropagatable gradients to the output space. A CKNOWLEDGMENTS
We thank the reviewers for providing their valuable time and insightful feedback.
A PPENDIX A NCE RANKING METHOD WITH s ( x , y ; Θ) = − E Θ ( x , y )
As pointed out by Ma & Collins (2018) as well, If we set s ( x , y ; Θ) = − E Θ ( x , y ) , we can regard this this learning with E (cid:48)
Θ ( x , y ) =
E Θ ( x , y ) + log P N
( y ) .
As pointed out earlier, this leads the model to estimate P ( y | x with
[Equation]
Plugging in E Θ (cid:48) as E Θ ( x , y ) + log P N ( y ) once again, we can see that P ( Y | X ) is estimated by [Equation]
B G RADIENT TO THE LABEL REPRESENTATIONS
As described in section 4, the energy component of loss L F , for the feedforward network, is the sum of following two parts:
[Equation]
and
[Equation]
The gradient of E global w.r.t y is given by the following vector of length
L
[Equation]
While the gradient of cross entropy l ( y, y ∗ ) w.r.t.
y is given by the following vector of length
L
[Equation]
(13) where ⊗ is element-wise product.
Note the dependency of the gradient for a particular component y i of y on other component in the expressions above.
The gradient of cross entropy ( ∂l∂y i ) only depends on its label dimension i : y i and y ∗ i .
In contrast, in the case of the energy function as loss (SEAL), the gradient ∂ L F ∂y i depends on all L dimensions of y due to the product term My in ∂E global ∂y i .
C C OMPARISON OF SEAL AND SEPARATE ENERGY NETWORK
In the following ﬁgures, we compare the F1 scores of SEAL framework with L E − regression (Figure 3),
L E − NCEranking (Figure 4) and L E − margin (Figure 5) on validation set with its energy-only counterpart.
We further report validation F1 scores for the output of GBI starting from the samples drawn from ˆ y and randomly.
We show this comparison to evaluate the performance of the energy network on points around ˆ y and random points.
[Figure]
[Figure] [Figure]
D H YPERPARAMETERS
The following tables tabulate the hyper-parameters for SEAL framework for each dataset.
We further ﬁx the number of layers, layer-dimensions and dropout in feed-forward network based on the conﬁguration of the best model trained using cross-entropy.
We ﬁx the λ 2 = 1 in equation 1 to reduce degree of freedom in hyperparameter search.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Here, we present the analysis on number of parameters (Table 11), inference speed (Table 12)) [Table] and training speed (Table 13) that different methods utilize per dataset.
In summary, SEAL requires approximately twice number of parameters compared to cross-entropy loss on training time, however it requires the same amount of parameter on inference time. Similarly, train time (sec) per epoch of SEAL is slower than that of cross-entropy and energy network as SEAL requires multiple backpropagation steps for both energy network and feedforward network. Nonetheless, again, at inference time
[Table]
the speed of CE and SEAL is the same and is much faster (2x-7x) than the inference of energy network. The analysis we present below are on the seven feature-based dataset, however the trend is similar for text-based dataset as well: twice parameter size on training time but equal parameter and runtime in inference time with CE method. The only difference in the text-based MLC is, due to running on a much larger BERT-baed models, that we only run one step for each update of feedforward network and energy network. Thus, unlike Table 13, train time of SEAL is approximately twice that of CE
[Table] [Table] [Table]