L EARNING A PPROXIMATE I NFERENCE N ETWORKS FOR S TRUCTURED P REDICTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Structured prediction energy networks (SPENs; Belanger & McCallum 2016) use neural network architectures to deﬁne energy functions that can capture arbitrary dependencies among parts of structured outputs.
Prior work used gradient descent for inference, relaxing the structured output to a set of continuous variables and then optimizing the energy with respect to them.
We replace this use of gradient descent with a neural network trained to approximate structured argmax inference.
This “inference network” outputs continuous values that we treat as the output structure.
We develop large-margin training criteria for joint training of the structured energy function and inference network.
On multi-label classiﬁcation we report speed-ups of 10-60x compared to (Belanger et al., 2017) while also improving accuracy.
For sequence labeling, for simple structured energies, our approach performs comparably to exact inference while being much faster at test time.
We also show how inference networks can replace dynamic programming for test-time inference in conditional random ﬁelds, suggestive for their general use for fast inference in structured settings.
I NTRODUCTION
Energy-based modeling (LeCun et al., 2006) associates a scalar measure of compatibility to each conﬁguration of input and output variables.
Given an input x , the predicted output ˆ y is chosen by minimizing an energy function E ( x , y ) .
For structured prediction, the parameterization of the energy function can leverage domain knowledge about the structured output space.
However, learning and prediction become complex.
Structured prediction energy networks (SPENs; Belanger & McCallum 2016) use an energy function to score structured outputs, and perform inference by using gradient descent to iteratively optimize the energy with respect to the outputs.
Belanger et al.
(2017) develop an “end-to-end” method that unrolls an approximate energy minimization algorithm into a ﬁxed-size computation graph that is trainable by gradient descent.
After learning the energy function, however, they still must use gradient descent for test-time inference.
We replace the gradient descent approach with a neural network trained to do inference, which we call an inference network .
It can have any architecture such that it takes an input x and returns an output interpretable as a y .
As in prior work, we relax y from discrete to continuous.
For multi-label classiﬁcation, we use a feed-forward network that outputs a vector.
We assign a single label to each dimension of the vector, interpreting its value as the probability of predicting that label.
For sequence labeling, we output a distribution over predicted labels at each position in the sequence.
We adapt the energy functions such that they can operate with both discrete ground truth outputs and outputs generated by our inference networks.
We deﬁne large-margin training objectives to jointly train the energy functions and inference networks.
They avoid argmax computations during training, making training faster than standard SPENs.
We experiment with multi-label classiﬁcation using the same setup as Belanger & McCallum (2016).
We show speed-ups of 10x in training time and 60x in test-time inference while also improving accuracy.
We then design a SPEN and inference network for sequence labeling by using recurrent neural networks (RNNs).
We perform comparably to a conditional random ﬁeld (CRF) when using the same energy function, with faster test-time inference.
We also experiment with a richer energy that includes a “label language model” that scores entire output label sequences using an RNN, showing it can improve handling of long-distance dependencies in part-of-speech tagging.
S TRUCTURED P REDICTION E NERGY N ETWORKS
We denote the space of inputs by X .
For a given input x ∈ X , we denote the space of legal structured outputs by Y ( x ) .
We denote the entire space of structured outputs by Y = ∪ x ∈X Y ( x ) .
A SPEN deﬁnes an energy function E Θ : X × Y → R parameterized by Θ that uses a functional architecture to compute a scalar energy for an input/output pair.
We describe the SPEN for multi-label classiﬁcation (MLC) from Belanger & McCallum (2016).
Here, x is a ﬁxed-length feature vector.
We assume there are L labels, each of which can be on or off for each input, so Y ( x ) = { 0 , 1 } L for all x .
The energy function is the sum of two terms:
E Θ
[Equation]
=
[Equation]
x
[Equation]
+
[Equation]
lab
[Equation]
y ) .
E loc ( x , y ) is the sum of linear models:
[Equation]
where b i is a parameter vector for label i and F ( x ) is a multi-layer perceptron computing a feature representation for the input x .
E lab ( y ) scores y independent of x :
[Equation]
where c 2 is a parameter vector, g is an elementwise non-linearity function, and C 1 is a parameter matrix.
After learning the energy function, prediction minimizes energy:
[Equation]
[Equation]
However, solving Eq.
(3) requires combinatorial algorithms because Y is a discrete structured space.
This becomes intractable when E Θ does not decompose into a sum over small “parts” of y .
Belanger & McCallum (2016) relax this problem by allowing the discrete vector y to be continuous.
We use
Y R to denote the relaxed output space.
For MLC, Y R ( x ) =
[0 , 1] L .
They solve the relaxed problem by using gradient descent to iteratively optimize the energy with respect to y .
Since they train with a structured large-margin objective, repeated inference is required during learning.
They note that using gradient descent for this inference step is time-consuming and makes learning less stable.
So
Belanger et al.
(2017) propose an “end-to-end” learning procedure inspired by Domke (2012).
This approach performs backpropagation through each step of gradient descent, permitting more stable training but also evidently more overﬁtting.
We compare to both methods in our experiments below.
I NFERENCE N ETWORKS FOR SPEN S
Belanger & McCallum (2016) relaxed y from a discrete to a continuous vector and used gradient descent for inference.
We also relax y
but we use a different strategy to approximate inference.
We deﬁne an inference network A Ψ ( x ) parameterized by Ψ and train it with the goal that
[Equation]
Given an energy function E Θ and a dataset X of inputs, we solve the following optimization problem:
[Equation]
The architecture of A Ψ will depend on the task.
For MLC, the same set of labels is applicable to every input, so y has the same length for all inputs.
So, we can use a feed-forward network for A Ψ with a vector output, treating each dimension as the prediction for a single label.
For sequence labeling, each x (and therefore each y ) can have a different length, so we must use a network architecture for A Ψ that permits different lengths of predictions.
We use an RNN that returns a vector at each position of x .
We interpret this vector as a probability distribution over output labels at that position.
We note that the output of A Ψ must be compatible with the energy function, which is typically deﬁned in terms of the original discrete output space Y .
This may require generalizing the energy function to be able to operate both on elements of Y and Y R .
For MLC, no change is required.
For sequence labeling, the change is straightforward and is described below in Section 7.2.1.
J OINT T RAINING OF SPEN S AND I NFERENCE N ETWORKS
Belanger & McCallum (2016) propose a structured hinge loss for training SPENs:
[Equation]
where D is the set of training pairs, [ f ] + = max(0 , f ) , and (cid:52) ( y , y (cid:48) ) is a structured cost function that returns a nonnegative value indicating the difference between y and y (cid:48) .
This loss is often referred to as “margin-rescaled” structured hinge loss (Taskar et al., 2004; Tsochantaridis et al., 2005).
However, this loss is expensive to minimize for structured models because of the “cost-augmented” inference step ( max y ∈Y R ( x ) ).
In prior work with SPENs, this step used gradient descent.
We replace this with a cost-augmented inference network A Φ ( x ) .
As suggested by the notation, the cost-augmented inference network A Φ and the inference network A Ψ will typically have the same functional form, but use different parameters Φ and Ψ .
We write our new optimization problem as:
[Equation]
We treat this optimization problem as a minimax game and ﬁnd a saddle point for the game.
Following
Goodfellow et al.
, we implement this using an iterative numerical approach.
We alternatively optimize Φ and Θ , holding the other ﬁxed.
Optimizing Φ to completion in the inner loop of training is computationally prohibitive and may lead to overﬁtting.
So we alternate between one mini-batch for optimizing Φ and one for optimizing Θ .
We also add L 2 regularization terms for Θ and Φ .
The objective for the cost-augmented inference network is:
[Equation]
That is, we update Φ so that A Φ yields an output that has low energy and high cost, in order to mimic cost-augmented inference.
The energy parameters Θ are kept ﬁxed.
There is an analogy here to the generator in generative adversarial networks (Goodfellow et al., 2014):
A Φ is trained to produce a high-cost structured output that is also appealing to the current energy function.
To help stabilize training of Φ , we add several terms to this objective, discussed below in Section 5.
The objective for the energy function is:
[Equation]
That is, we update Θ so as to widen the gap between the cost-augmented and ground truth outputs.
There is an analogy here to the discriminator in generative adversarial networks.
The energy function is updated so as to enable it to distinguish “fake” outputs produced by A Φ from real outputs
y i .
Training iterates between updating Φ and Θ using the objectives above.
4. T EST -T IME I NFERENCE
After training, we want to use an inference network A Ψ deﬁned in Eq.
However, training only gives us a cost-augmented inference network A Φ .
Since A Ψ and A Φ have the same functional form, we can use Φ to initialize Ψ , then do additional training on A Ψ as in Eq.
(5) where X is the training or validation set.
This step helps the resulting inference network to produce outputs with lower energy, as it is no longer affected by the cost function.
Since this procedure does not use the output labels of the x ’s in X , it could also be applied to the test data in a transductive setting.
4. V ARIATIONS AND S PECIAL C ASES
This approach also permits us to use large-margin structured prediction with slack rescaling (Tsochantaridis et al., 2005).
Slack rescaling can yield higher accuracies than margin rescaling, but requires “cost-scaled” inference during training which is intractable for many classes of output structures.
However, we can use our notion of inference networks to circumvent this tractability issue and approximately optimize the slack-rescaled hinge loss, yielding the following optimization problem:
[Equation]
Using the same argument as above, we can also break this into alternating optimization of Φ and Θ .
We can optimize a structured perceptron (Collins, 2002) version by using the margin-rescaled hinge loss (Eq.
and ﬁxing (cid:52) ( A Φ ( x i ) , y i )
= 0 .
When using this loss, the cost-augmented inference network is actually a test-time inference network, because the cost is always zero, so using this loss may lessen the need to retune the inference network after training.
When we ﬁx (cid:52) ( A Φ ( x i ) , y i ) = 1 , then margin-rescaled hinge is equivalent to slack-rescaled hinge.
While using (cid:52) = 1 is not useful in standard max-margin training with exact argmax inference (because the cost has no impact on optimization when ﬁxed to a positive constant), it is potentially useful in our setting.
Consider our SPEN objectives with (cid:52) = 1 :
[Equation]
There will always be a nonzero difference between the two energies because A Φ ( x i ) will never exactly equal the discrete vector y i .
Since there is no explicit minimization over all discrete vectors y , this case is more similar to a “contrastive” hinge loss which seeks to make the energy of the true output lower than the energy of a particular “negative sample” by a margin of at least 1.
In our experiments, we will compare four hinge losses for training SPENs: margin-rescaled (Eq.
(7)), slack-rescaled (Eq.
, perceptron (margin-rescaled with (cid:52) = 0 ), and contrastive ( (cid:52) = 1 ).
I MPROVING T RAINING FOR I NFERENCE N ETWORKS
We found that the alternating nature of the optimization led to difﬁculties during training.
Similar observations have been noted about other alternative optimization settings, especially those underlying generative adversarial networks (Salimans et al., 2016).
Below we describe several techniques we found to help stabilize training, which are optional terms added to the objective in Eq.
L 2 Regularization: We use L 2 regularization, adding the penalty term (cid:107) Φ (cid:107) 22 with coefﬁcient λ 1 .
Entropy Regularization: We add an entropy-based regularizer loss H ( A Φ ( x )) deﬁned for the problem under consideration.
For MLC, the output of A Φ ( x ) is a vector of scalars in [0 , 1] , one for each label, where the scalar is interpreted as a label probability.
The entropy regularizer loss H is the sum of the entropies over these label binary distributions.
For sequence labeling, where the length of x is N and where there are L unique labels, the output of A Φ ( x ) is a length-
N sequence of length- L vectors, each of which represents the distribution over the L labels at that position in x .
Then, loss H is the sum of entropies of these label distributions across positions in the sequence.
When tuning the coefﬁcient λ 2 for this regularizer, we consider both positive and negative values, permitting us to favor either low- or high-entropy distributions as the task prefers.
Local Cross Entropy Loss: We add a local (non-structured) cross entropy loss
CE
( A Φ ( x i ) , y i ) deﬁned for the problem under consideration.
We only experiment with this loss for sequence labeling.
It is the sum of the label cross entropy losses over all positions in the sequence.
This loss provides more explicit feedback to the inference network, helping the optimization procedure to ﬁnd a solution that minimizes the energy function while also correctly classifying individual labels.
It can also be viewed as a multi-task loss for the inference network.
Regularization Toward Pretrained Inference Network: We add the penalty (cid:107) Φ − Φ 0 (cid:107) 2 2 where Φis a pretrained network, e.g., a local classiﬁer trained to independently predict each part of y .
We only experiment with this regularizer for the MLC tasks.
Each additional term has its own tunable hyperparameter.
Finally we obtain:
[Equation]
R ELATED W ORK
Our methods are reminiscent of other alternating optimization problems like that underlying generative adversarial networks (GANs; Goodfellow et al. 2014).
GANs are based on a minimax game and have a value function that one agent (a discriminator D ) seeks to maximize and another (a generator
G ) seeks to minimize.
By their analysis, a log loss discriminator converges to a degenerate uniform solution.
When using hinge loss, we can get a non-degenerate discriminator while matching the data distribution (Dai et al., 2017).
Our formulation is closer to this hinge loss version of the GAN.
Our approach is also related to knowledge distillation (Ba & Caruana, 2014; Hinton et al., 2015), which refers to strategies in which one model (a “student”) is trained to mimic another (a “teacher”).
Typically, the teacher is a larger, more accurate model but which is too computationally expensive to use at test time.
Urban et al.
(2016) train shallow networks using image classiﬁcation data labeled by an ensemble of deep teacher nets.
Geras et al.
(2016) train a convolutional network to mimic an
LSTM for speech recognition.
Others have explored knowledge distillation for sequence-to-sequence learning (Kim & Rush, 2016) and parsing (Kuncoro et al., 2016).
Since we train a single inference network for an entire dataset, our approach is also related to “amortized inference” (Srikumar et al., 2012; Gershman & Goodman, 2014; Paige & Wood, 2016).
These methods precompute or save solutions to subproblems for faster overall computation.
It is likely that inference networks devote more modeling capacity to the most frequent substructures in the data.
A kind of inference network is used in variational autoencoders (Kingma & Welling, 2013) to approximate posterior inference for training generative models.
Our methods are also related to work in structured prediction that seeks to approximate structured models with factorized ones, e.g., mean-ﬁeld approximations in graphical models (Koller & Friedman, 2009; Krähenbühl & Koltun, 2011).
There are other settings in which gradient descent is used for inference, e.g., image generation applications like DeepDream (Mordvintsev et al., 2015) and neural style transfer (Gatys et al., 2015), as well as machine translation (Hoang et al., 2017).
In these and related settings, gradient descent has started to be replaced by inference networks.
Our results below provide more evidence for making this transition.
An alternative to what we pursue here would be to obtain an easier convex optimization problem for inference via input convex neural networks (Amos et al., 2017).
E XPERIMENTS
In Sec.
7.1 we compare our approach to previous work on training SPENs for MLC.
We compare accuracy and speed, ﬁnding our approach to outperform prior work.
We then perform experiments with sequence labeling tasks in Sec. 7.2.
7. M ULTI -L ABEL C LASSIFICATION
We use the MLC datasets used by Belanger & McCallum (2016): Bibtex, Delicious, and Bookmarks.
Dataset statistics are shown in Table 8 in the Appendix.
For Bibtex and Delicious, we follow Belanger and McCallum and tune the hyperparameters using a different sampling of train and test data, then use the standard train/test split for ﬁnal experimentation using the tuned hyperparameters.
For Bookmarks, we use the same train/dev/test split as (Belanger & McCallum, 2016).
For evaluation, we report the example averaged (macro averaged) F1 measure.
We use the SPEN for MLC described in Section 2 and also used by Belanger & McCallum (2016).
We pretrain the feature networks F ( x ) by minimizing independent-label cross entropy for 10 epochs
[Table] [Table]
using Adam (Kingma & Ba, 2014) with learning rate 0.001.
While training SPENs, we only update the parameters of the energy function ( Θ ) and the inference network ( Φ ), keeping the feature network parameters F ( x ) ﬁxed.
We use Adam with learning rate 0.001 to train Θ and Φ .
We tune λ (the Lregularization strength for Θ ) over the set { 0 . 01 , 0 .
The inference networks are feed-forward networks with two hidden layers, using the same architectures as the feature networks F ( x ) .
This permits us to initialize inference network parameters Φ using pretrained feature network parameters.
For the output, we use an afﬁne transformation layer with a sigmoid nonlinearity function, so the output values are in the range (0 , 1) .
We interpret each value as the probability of predicting the corresponding label.
We obtain discrete predictions by thresholding at a threshold τ tuned to maximize F1 on the development data.
3 We add three terms to the inference network objective from Section 5: L 2 regularization ( λ 1 ∈ { 0 . 01 , 0 . 001 , 0 . 0001 } ), entropy regularization ( λ 2 = 1 ), and regularization toward the pretrained feature network ( λ 4 ∈ { 0 , 1 , 10 } ).
Comparison of Loss Functions and Impact of Inference Network Retuning.
We ﬁrst report results comparing the four loss functions from Section 4.2.
Table 1 shows results on the development set for Bookmarks, the largest of the three datasets.
We ﬁnd performance to be highly similar across the losses, with the contrastive loss appearing slightly better than the others.
After training, we “retune” the inference network as speciﬁed by Eq.
(5) on the development set for 20 epochs using a smaller learning rate of 0.00001.
Table 1 shows slightly higher F1 for all losses with retuning.
We were surprised to see that the ﬁnal cost-augmented inference network performs well as a test-time inference network.
This suggests that by the end of training, the cost-augmented network may be approaching the argmin and that there may not be much need for retuning.
When using (cid:52) = 0 or 1, retuning leads to the same small gain as when using the margin-rescaled or slack-rescaled losses.
Here the gain is presumably from adjusting the inference network for other inputs rather than from converting it from a cost-augmented to a test-time inference network.
Comparison to Prior Work.
Table 2 shows results comparing to prior work.
The MLP and
SPEN baseline results are taken from (Belanger & McCallum, 2016).
We obtained the “SPEN (E2E)”
(Belanger et al., 2017) results by running the code available from the authors on these datasets.
This method constructs a recurrent neural network that performs gradient-based minimization of the energy with respect to y .
They noted in their software release that, while this method is more stable, it is prone to overﬁtting and actually performs worse than the original SPEN.
We indeed ﬁnd this to be the case, as SPEN (E2E) underperforms SPEN on all three datasets.
Our method uses the contrastive hinge loss and retunes the inference network; these decisions were made based on the results in the previous section.
Our approach achieves the best average performance across the three datasets.
It performs especially well on Bookmarks, which is the largest of the three.
[Table]
[Table]
[Table]
Speed Comparison.
Table 3 compares training and test-time inference speed among the different methods.
We only report speeds of methods that we ran.
The SPEN (E2E) times were obtained using code obtained from Belanger and McCallum.
We suspect that SPEN training would be comparable to or slower than SPEN (E2E).
Our method can process examples during training about 10 times as fast as the end-to-end SPEN, and 60-130 times as fast during test-time inference.
In fact, at test time, our method is roughly the same speed as the MLP baseline, since our inference networks use the same architecture as the feature networks which form the MLP baseline.
Compared to the MLP, the training of our method takes signiﬁcantly more time overall because of joint training of the energy function and inference network, but fortunately the test-time inference is comparable.
7. S EQUENCE L ABELING
We also evaluate our methods on sequence labeling.
We perform experiments with Twitter part-ofspeech (POS) tagging here.
Named entity recognition experiments are reported in the Appendix.
7.2. E NERGY F UNCTIONS FOR S EQUENCE L ABELING
The input space X is now the set of all sequences of symbols drawn from a vocabulary.
For an input sequence x of length N , where there are L possible output labels for each position in x , the output space Y ( x ) is [ L ] N , where the notation [ q ] represents the set containing the ﬁrst q positive integers.
We deﬁne y =
(cid:104)
y 1 , y 2 , .., y N (cid:105) where each y i ranges over possible output labels, i.e., y i ∈
[ L ] .
When deﬁning our energy for sequence labeling, we take inspiration from bidirectional LSTMs (BLSTMs; Hochreiter & Schmidhuber 1997) and conditional random ﬁelds (CRFs; Lafferty et al.
A “linear chain” CRF uses two types of features: one capturing the connection between an output label and x and the other capturing the dependence between neighboring output labels.
We use a BLSTM to compute feature representations for x .
We use f ( x , t ) ∈ R d to denote the “input feature vector” for position t , deﬁning it to be the d -dimensional BLSTM hidden vector at t .
We then deﬁne the following energy function:
[Equation]
where U i ∈
R
d is a parameter vector for label
i and the parameter matrix
W ∈ R L × L contains label pair parameters.
The full set of parameters Θ includes the U i vectors, W , and the parameters of the
BLSTM.
The above energy only permits discrete y .
For the general case that permits relaxing y to be continuous, we treat each y t as a vector.
It will be one-hot for the ground truth y and will be a vector of label probabilities for relaxed y ’s.
Then the general energy function is:
[Equation]
where y t,i is the i th entry of the vector y t .
In the discrete case, this entry is 1 for a single i and 0 for all others, so this energy reduces to Eq.
(14) in that case.
In the continuous case, this scalar indicates the probability of the t th position being labeled with label i .
For the label pair terms in this general energy function, we use a bilinear product between the vectors y t − 1 and y t using parameter matrix
W , which also reduces to Eq.
(14) when they are one-hot vectors.
Below we refer to Eq.
(15) as the “pairwise energy”
and we refer to a version of it with only the unary label terms as the “local energy”.
[Table] [Table]
Tag Language Model.
In order to capture long-distance dependencies in an entire sequence of labels
[Table]
we train a “tag language model” on a large corpus of automatically-tagged tweets, then include a term in the energy function representing the log-probability of the given tag sequence under this tag language model.
Full details are included in the Appendix.
7.2. E XPERIMENTAL S ETUP
For Twitter part-of-speech (POS) tagging, we use the annotated data from Gimpel et al.
(2011) and Owoputi et al.
(2013) which contains L = 25 POS tags.
For training [Table] we combine the 1000tweet O CT 27T RAIN set and the 327-tweet O CT 27D EV set.
For validation, we use the 500-tweet O CT 27T EST set and for testing we use the 547-tweet D AILY 547 test set.
We pretrain 100-dimensional skip-gram word embeddings on 56 million English tweets using word2vec (Mikolov et al., 2013).
Details for training the tag language model are provided in the Appendix.
We compare to a BLSTM baseline which consists simply of the local energy terms in the energy function.
We use hidden vectors of dimensionality d = 100 .
For our method, we use BLSTMs for the inference networks.
The output layer of the inference network is a softmax function, so at every position, the inference network produces a distribution over labels at that position.
During training, we use early stopping based on validation accuracy.
For (cid:52) , we use L 1 distance.
We train inference networks using stochastic gradient descent (SGD) with momentum and train the energy parameters using Adam.
We tune hyperparameters on the validation set; full details of tuning are provided in the Appendix.
We train CRF baselines with the standard CRF training objective (conditional log-likelihood) using the standard dynamic programming algorithms (forward-backward) to compute gradients during training.
For optimization we use SGD with momentum.
7.2. R ESULTS
Comparison to Exact Inference at Training and Test Time.
In Table 4, we report results for
BLSTM and CRF baselines.
We then train inference networks given the ﬁxed CRF energy function with various stabilization terms.
When using either entropy or cross entropy, our inference networks outperform Viterbi. When using the squared L 2 distance term which regularizes the inference network toward the pretrained BLSTM, the accuracy reduces to be very close to that of the BLSTM. [Table]
[Table]
When using the CRF energy function, we can compare the use of exact and approximate inference during training.
The lower part of Table 4 shows results when training a SPEN with the CRF energy function. We ﬁnd that, despite using approximate inference during training (i.e., jointly training the inference network and the energy function), the SPEN reaches comparable accuracies to Viterbi on the development set, at least when using cross entropy as an additional stabilizing term.
Loss Function Comparison.
Table 5 shows results when comparing SPEN training objectives.
We see a larger difference among losses here than for MLC tasks.
When using the perceptron loss, there is no margin, which leads to overﬁtting (89.4 on dev, 88.6 on test).
The contrastive loss, which strives to achieve a margin of 1, does better on test (89.0).
We also see here that margin rescaling and slack rescaling both outperform the contrastive hinge, unlike the MLC tasks.
We suspect that in the case in which each input/output has a different length, using a cost that captures length is more important.
Tag Language Models.
The above results only use the pairwise energy; no results used the tag language model (TLM).
Table 6 shows results when using the TLM as an additional term in training the inference network for the CRF energy, where we tune the weight of the TLM term over the set { 0 . 1 , 0 . 2 , 0 . 5 } .
We ﬁnd a gain of 0.6 on the test set.
We investigated the improvements and found some to involve corrections that seemingly stem from handling non-local dependencies better.
Tablein the Appendix shows examples in which the model with the tag language model appears to be better at using the broader context when making tagging decisions.
These results suggest that our method of training inference networks can be used to add rich global features to structured prediction, though we leave a thorough exploration of features to future work.
Speed Comparison.
Table 7 compares the training and test-time inference speed of several methods.
While our method is slower than the baselines during training, it is faster than the CRF at test time, operating at essentially the same speed as the BLSTM baseline while being more accurate.
C ONCLUSIONS AND F UTURE W ORK
We presented ways to jointly train structured energy functions and inference networks using largemargin objectives.
The energy function captures arbitrary dependencies among the labels, while the inference networks learns to capture the properties of the energy in an efﬁcient manner, yielding fast test-time inference.
Future work includes exploring the space of network architectures for inference networks to balance accuracy and efﬁciency, experimenting with additional global terms in structured energy functions, and exploring richer structured output spaces such as trees and sentences.
A PPENDIX 9. M ULTI -L ABEL C LASSIFICATION
Table 8 shows dataset statistics for the multi-label classiﬁcation datasets.
9. T WITTER POS T AGGING
Figure 1 shows the learned pairwise potential matrix W in Twitter POS tagging.
We can see strong correlations between labels in neighborhoods.
For example, an adjective (A) is more likely to be followed by a noun (N) than a verb (V) (see row labeled “A” in the ﬁgure) [Table]
9.2. H YPERPARAMETER T UNING
When training inference networks and SPENs for Twitter POS tagging, we use the following hyperparameter tuning.
We tune the inference network learning rate ( { 0 . 1 , 0 .
L 2 regularization ( λ 1 ∈ { 0 , 1e − 3 , 1e − 4 , 1e − 5 , 1e − 6 , 1e − 7 } ), the entropy regularization term ( λ 2 ∈ { 0 . 1 , 0 . 5 , 1 , 2 , 5 , 10 } ), the cross entropy regularization term ( λ 3 ∈ { 0 . 1 , 0 .
5 , 1 , 2 , 5 , 10 } ), and the squared L2 distance ( λ 4 ∈ { 0 , 0 . 1 , 0 . 2 , 0 .
We train the energy functions with Adam with a learning rate of 0.001 and L 2 regularization ( λ 1 ∈ { 0 , 1e − 3 , 1e − 4 , 1e − 5 , 1e − 6 , 1e − 7 } ).
9.2. T AG L ANGUAGE M ODEL D ESCRIPTION
Let Y t denote the random variable for the t th label in the sequence Y with length m , y t denote the value of Y t and the distributed representation (embedding) of this label.
[Equation]
Where y 0 is the start symbol, y m +1 is the end symbol, W and b are the parameters of the model,
LSTM is a long short-term memory network (Hochreiter & Schmidhuber, 1997), and h t is the hidden unit of t -th label.
When training the tag language model, y i is a one-hot feature vector including the sequence end symbol.
[Table]
[Table]
we move urgently needed #food [Table] wheat
[Table]
by truck convoy into western libya for 1st time
[Table]
... how can people choose there favourite performance , the whole thing was perfect !
[Table]
After training the tag language model, we can compute the probability for any tag sequence, and we deﬁne the following term:
[Equation]
where y j is a softmax distribution over tags for position j , A Ψ ( x ) m +1 is the end-of-sequence symbol, ∗ is the inner product operation, and TLM is the function that computes the sequence of softmax distributions using the pretrained tag language model.
We tune the relative weight for this extra energy term E TLM over the set { 0 . 1 , 0 . 2 , 0 . 5 } .
For this tag language model feature, we automatically tag unlabeled tweets, then train the LSTM language model on the automatic tag sequences.
In particular, we run the Twitter POS tagger from
Owoputi et al.
on a dataset of 303K randomly-sampled English tweets, using 300K for training and 3K for tuning hyperparameters and early stopping when training this tag language model.
We train an LSTM language model on the tag sequences using stochastic gradient descent with momentum and early stopping on the validation set.
We used a dropout rate of 0.5 for the LSTM hidden layer.
We tune the learning rate ( { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 } ), the number of LSTM layers ( { 1 , 2 } ), and the hidden layer size ( { 50 , 100 , 200 } ).
Table 9 shows examples in which our model with the global energy term (the tag language model) appears to be using broader context when making tagging decisions.
Here we are comparing predictions in the test set between two settings in Table 6: “CRF, Inference network (trained w/ entropy)” vs. “CRF, Inference network (trained w/ entropy + TLM)”.
In example 1, the token “#food” is tagged as a hashtag by the model without the TLM.
However, it is annotated as a noun in the gold standard because it is functioning as a noun in this context.
If only shown “urgently needed #food (”, it is unclear whether “#food” is being used as a hashtag or as a noun in the tweet.
Example 3 similarly beneﬁts from the broader context, but this time in an adjective/noun ambiguity.
In example 2, the model with the TLM is able to recognize that “there”, while appearing to be an adverb based on its surface form, is actually a misspelling of the determiner “their” based on the context.
9. N AMED E NTITY R ECOGNITION
For named entity recognition (NER), we perform experiments on the English data from the CoNLL 2003 shared task (Tjong Kim Sang & De Meulder, 2003).
This task contains sentences annotated with named entities and their types, containing 14987 training sentences, 3466 in the development set, and 3684 in the test set.
There are four named entity types: PERSON, LOCATION, ORGANIZATION, and MISC.
We use the BIOES tagging scheme instead of the original BIO2, following prior work (Ratinov & Roth, 2009; Ma & Hovy, 2016).
There are L = 17 classes.
We use 100-dimensional pretrained GloVe Pennington et al.
(2014) embeddings trained on 6 billion words from Wikipedia and web text, which work better than other pretrained embeddings (Ma & Hovy, 2016).
Results are shown in Table 10.
We see a large 4-point gap between the BLSTM and CRF, suggesting the importance of structured information for this problem.
When training inference networks for the [Table] trained CRF (rows labelled “CRF + InfNet”), we ﬁnd that we need multiple layers in the inference network in order to approach the result of the Viterbi algorithm.
However, the 1-layer inference network result of 84.52 F1 is as fast in test-time inference as the BLSTM baseline which yields 83.
F1.
Also, while the SPEN losses underperform the CRF with Viterbi, they slightly outperform the
CRF with the 1-layer inference network while having the same test-time inference speed.