Measuring and Reducing Model Update Regression in Structured Prediction for NLP
Abstract
Recent advance in deep learning has led to the rapid adoption of machine learningbased NLP models in a wide range of applications.
Despite the continuous gain in accuracy, backward compatibility is also an important aspect for industrial applications, yet it received little research attention.
Backward compatibility requires that the new model does not regress on cases that were correctly handled by its predecessor.
This work studies model update regression in structured prediction tasks.
We choose syntactic dependency parsing and conversational semantic parsing as representative examples of structured prediction tasks in NLP.
First, we measure and analyze model update regression in different model update settings.
Next, we explore and benchmark existing techniques for reducing model update regression including model ensemble and knowledge distillation.
We further propose a simple and effective method, Backward-Congruent Re-ranking (BCR), by taking into account the characteristics of structured prediction.
Experiments show that BCR can better mitigate model update regression than model ensemble and knowledge distillation approaches.
Introduction
Model update regression refers to the deterioration of performance in some test cases after a model update, even if the new model has on average better performance than the old model.
For example, on a ﬁxed test set, the new model may achieve 1% absolute improvement in accuracy.
However, it does not necessarily mean that the new model gets 1% extra test cases correct.
Instead, it may correct 6% mistakes made by its predecessor but also introduce 5% new errors.
The occasional worse behavior of the new model can overshadow the beneﬁts of overall performance gains and hinder its adoption.
Imagine a newly updated virtual assistant stops understanding a user’s favorite ways to inquire about the local trafﬁc.
This can severely degrade the user experience even if the assistant has been improved in other aspects.
Prior research (Shen et al., 2020; Yan et al., 2021; Xie et al., 2021) has investigated model update regression in classiﬁcation problems of computer vision and natural language processing (NLP).
However, the formalization of classiﬁcation only has limited coverage of real-world NLP applications.
For instance, the natural language understanding component behind modern virtual assistants aims to transform user utterances into semantic graphs, a task known as conversational semantic paring
(Figure 1).
In contrast to classiﬁcation tasks, the global prediction of structured prediction (e
[Figure]
g., graph or tree) is often composed of many local predictions (e
[Figure]
g., nodes and edges).
Some local predictions can be correct while the global prediction is imperfect.
Therefore, model update regression can happen at ﬁne-grained levels.
Also, the output space is input-dependent and can be extremely large.
To our knowledge
[Figure]
the model update regression problem has not been studied in structured prediction tasks
[Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
party‘s
Directions to David’s party.
Old Model
New Model model update Output:
Input:
IN:GET_EVENT
SL:DESTINATION IN:FIND_EVENT
SL:LOCATION
SL:CATEGORY directions to
Davidparty‘s
In this work, we measure and analyze model update regression in two typical NLP structured prediction tasks: one general-purpose task (i.e., syntactic dependency parsing) and one application-oriented task (i.e., conversational semantic parsing).
We set up evaluation protocols and test a range of model updates rooted in different aspects, including changes in model design, training data, and optimization process.
Experiments show that model update regression is prevalent and evident across various model update settings.
The above ﬁnding shows a general and critical demand for techniques to reduce model update regression in structured prediction.
Prior work for classiﬁcation problems (Shen et al., 2020; Yan et al., 2021;
Xie et al., 2021) has borrowed the idea of knowledge distillation (Hinton et al., 2015), where the new model (student) is trained to ﬁt the output distribution of the old model (teacher).
However, vanilla knowledge distillation (Hinton et al., 2015) cannot be directly applied to structured prediction models due to the intractability of the exact distribution of global prediction.
Moreover, the goal prediction of structured prediction can be factorized in distinct ways.
For example, a graph-based dependency parser (e.g., Dozat and Manning, 2017) scores every possible edge and searches for a maximum spanning tree, while a transition-based dependency parser (e.g., Ma et al., 2018) builds the tree incrementally through a series of actions.
We refer to a model update between models with different factorizations as a heterogeneous model update.
For heterogeneous model updates, existing factorization-speciﬁc approximation (e.g., Zmigrod et al., 2021a; Wang et al., 2021) or even knowledge distillation at the level of local predictions is inapplicable.
To develop generic solutions that do not assume any speciﬁc factorizations, we instead resort to the sequence-level knowledge distillation (Kim and Rush, 2016), which is originally proposed for machine translation.
Previous work also ﬁnds model ensemble, though not as practical due to high computational cost, a strong baseline for reducing model update regression.
We also include it for comparison purpose.
We further propose a novel and generic method named Backward-Congruent Re-ranking (BCR).
BCR takes into account the variety of structured prediction output.
That is, a new model can produce a set of different predictions achieving similar accuracies.
The key is to pick the one with the highest backward compatibility.
In a nutshell, BCR uses the old model as a re-ranker to select a top structure from a set of candidates predicted by the new model.
We further propose dropout- p sampling, a simple and generic sampling method, to improve the diversity and quality of candidates.
Despite the simplicity of BCR, it is a generic and surprisingly effective approach for mitigating model update regression, substantially outperforming knowledge distillation and ensemble methods across all studied model update settings.
More surprisingly, we ﬁnd that BCR can also improve the accuracy of the new model.
Related Work
Forgetting in Machine Learning Model update regression is related to a spectrum of research related to forgetting in machine learning, including continual learning (Chen and Liu, 2018; Parisi et al., 2019; Biesialska et al., 2020), incremental learning (Polikar et al., 2001; Gepperth and Hammer, 2016; Masana et al., 2020), which are concerned with the learning of new tasks or streaming examples and the forgetting of previously learned ones.
The model update regression problem differs in that models are tested on the same task and have full access to previous data.
Prior Research in Model Update Regression
The model update regression problem was ﬁrst studied in (Shen et al., 2020) on learning backward-compatible visual embeddings for image retrieval.
(Yan et al., 2021) suggested mitigating regression in image classiﬁcation with positive-congruent training, a specialized variant of knowledge distillation.
(Xie et al., 2021) extended the study of backward compatibility to NLP classiﬁcation and explored several knowledge distillation variants for regression reduction.
(Träuble et al., 2021) introduced a probabilistic approach to switch between the legacy and updated systems, bringing down regression in image classiﬁcation.
Our work investigates the model update regression in NLP structured prediction tasks, which can be particularly challenging due to the enormous output space and the variety of different factorizations.
Re-ranking in Structure Prediction The re-ranking technique has a long history in NLP for improving not only accuracy (Shen et al., 2004; Collins and Koo, 2005; Salazar et al., 2019; Yee et al., 2019), but also output diversity (Li et al., 2016), ﬂuency (Kriz et al., 2019; Yin and Neubig, 2019), fairness (Geyik et al., 2019), and factual consistency (Falke et al., 2019).
Different from prior work, we show that re-ranking can be adopted for improving backward compatibility in model updates.
Preliminaries
We ﬁrst lay out the basic concepts for the quantitative measurement of model update regression in structured prediction.
Then we introduce two representative structured prediction tasks in NLP: dependency parsing and conversational semantic parsing, and set up the evaluation benchmarks.
3.1 Prediction Flips in Structured Prediction
Model update regression happens after a model update , where an old model is replaced by a new model .
For typical NLP structured prediction tasks (Smith, 2011), the input is a piece of natural language text and the output is a graph G =
( V, E ) with multiple nodes V and edges E .
Unlike classiﬁcation problems where the output is a label in a pre-deﬁned label set, the output graph G is decomposable and we may be interested in the correctness of some meaningful sub-graph G {+
+}
= ( V 0 ✓ V, E 0 ✓ E ) even though the graph is not completely correct.
We call the graphs (or sub-graphs) that are correctly predicted by both the new and the old models as positive-congruent predictions.
On the other hand, negative ﬂips are the cases where the graphs (or sub-graphs) are correctly predicted by the old model but incorrectly predicted by the new one.
There are also positive ﬂips , where the new model corrects the mistakes that the old one makes, which are desirable accuracy improvements.
Negative Flip Rate (NFR)
To measure the severity of model update regression, we count negative ﬂips and compute their fraction of the total number of graphs (or sub-graphs).
We term the fraction as negative ﬂip rate (NFR) (Yan et al., 2021).
Negative Flip Impact (NFI) Note that NFR is capped by the overall error rate (ER) of the new model, comparison is challenging across cases with different ERs.
For this reason, we introduce the negative ﬂip impact (NFI): NFI =
NFR / ER new .
Intuitively, the NFI computes the proportion of errors caused by negative ﬂips.
In other words, the proportion of errors that could have been avoided without the model update.
3.2 Dependency Parsing
The task of dependency parsing (Kübler et al., 2009) seeks to ﬁnd the syntactic head word for each word in a sentence and their syntactic relation.
The overall output has a tree structure where nodes are words in the sentence and edges are their relations.
The importance of dependency parsing is widely recognized in the NLP community, with it beneﬁting a wide range of NLP applications (Qi et al., 2020).
We use the English EWT treebank from the Universal Dependency (UD2.2) Treebanks.
2 We adopt the standard training/dev/test splits and use the universal POS tags (Petrov et al., 2012) provided in the treebank.
The parsing performance is measured by word-level accuracies: unlabeled/labeled attachment score (UAS/LAS), and tree-level accuracies: unlabeled/labeled complete match (UCM/LCM).
We compute NFR and NFI under each accuracy metric.
We ﬁnd that the results of labeled metrics are always consistent with unlabeled metrics.
For clarity, we only show the results of unlabeled metrics, the results of labeled metrics are presented in Appendix A.
Following conventions (Dozat and Manning, 2017; Ma et al., 2018), we report results excluding punctuations.
3.3 Conversational Semantic Parsing
Conversational semantic parsing is an important component for intelligent dialog systems such as
Amazon Alexa, Apple Siri, and Google Assistant, which transform user utterances into semantic frames comprised of intents and slots.
We use the TOP dataset (Gupta et al., 2018) for our experiments, which uses hierarchical intent-slot annotations for utterances in navigation and event domains.
The output is a constituency-based parse tree, where leaf nodes are words and the interior nodes are labeled with slots and intents.
We use the dataset version with the noisy IN:UNSUPPORTED_* intents excluded (Einolghozati et al., 2019).
The parsing performance is conventionally measured by tree-level Exact Match accuracy (EM).
We also report span-level EM accuracy (span EM):
We represent the semantic annotations on a sentence as a set of labeled spans ( i, j, L ) where i, j mark the start and end position of a span and L is the corresponding label.
For example, the gold tree in Figure 1 can be viewed as four labeled spans including (0, 4, IN:GET_DIRECTION ), (2, 4, SL:DESTINATION;IN:FIND_EVENT ), (2, 2, SL:ORGANIZER ), and (4, 4, SL:CATEGORY ).
We calculate the label accuracy on such spans.
For example, the new model’s output correctly predicted three of them, thus the span-level EM accuracy is 3 / 4 = 75% .
We calculate NFR and NFI under both metrics.
Measuring Model Update Regression
We carry out a series of experiments to examine the severity of model update regression under various model update scenarios.
In this section, we report the empirical results and conclude the ﬁndings.
4.1 Model Update Settings
In total, we consider four different
causes for model updates.
• Change the training conﬁgurations (e.g., random seed, learning rate, etc).
Despite this setting being less likely in practice, we consider it as a synthetic evaluation to understand the severity of model update regression.
• Scale up model sizes (e.g., increasing the depth or width of neural network).
It is common that using a larger pre-trained language model brings in a larger performance gain.
• Train on more data.
Practitioners frequently collect more and more labeled data for training.
• Take another factorization choice.
It happens when new algorithms or models are invented and adopted.
4.2 Dependency Parsing
Setup For dependency parsing, we consider two popular models, namely the biafﬁne parser (Dozat and Manning, 2017) ( deepbiaf ) and the stack-pointer parser (Ma et al., 2018) ( stackptr ).
The biafﬁne parser is a ﬁrst-order graph-based parser, where the global prediction is decomposed to arc/edge predictions between any two words.
The stack-pointer parser is a transition-based auto-regressive parser, where the global prediction is decomposed into a series of building action predictions.
These two represent two distinct factorizations of the dependency parsing problem.
Table 1 (top) presents their accuracies (mean and standard deviation).
We can see that in terms of word-level accuracy, the performances of deepbiaf and stackptr are very close.
However, in terms of complete match, stackptr is slightly better than deepbiaf , likely due to its auto-regressive nature.
We study model update regression in three different model update settings ( deepbiaf ) deepbiaf , stackptr ) stackptr , and deepbiaf ) stackptr ).
In the ﬁrst two settings, the old and new models have
[Table] [Table]
identical model architecture and training data.
The only difference is that they are trained with different random seeds.
We present these controlled experiments to demonstrate that even small changes such as altering random seeds can introduce signiﬁcant regression.
Results Table 1 (bottom) shows that model update regression is severe across all model update settings, including deepbiaf ) deepbiaf and stackptr ) stackptr .
This implies that it could be fundamentally difﬁcult to reduce model upgrade regression as initialization and optimization processes naturally introduce prediction inconsistencies.
On the other hand, model update regression in heterogeneous model update ( deepbiaf ) stackptr ) is even more severe.
Particularly, negative ﬂips account for up to 25% of the errors made by new models ( deepbiaf ) stackptr according to UAS).
Notably, despite that stackptr improves over deepbiaf by 1 .
95% on UCM, both NFR and NFI in deepbiaf ) stackptr is higher than those in homogeneous model updates.
This demonstrates that reducing the error rate is not sufﬁcient to reduce regression.
4.3 Conversational Semantic Parsing
Setup For conversational semantic parsing, we use the seq2seq parser (Rongali et al., 2020) in our experiments.
It formulates the semantic parsing task as seq2seq generation of the linearized parse tree given the source text.
The global prediction is thus decomposed into a sequence of next-word predictions.
The seq2seq parsers can be initialized with different pre-trained language models.
We experiment with two model variants: seq2seq parser with (1) roberta-base (Liu et al., 2019) ( s2s-base ) and (2) with roberta-large ( s2s-large ).
In order to simulate data-driven model updates, we also train parsers with only 4/5 of the available training data, denoted by s2s-base -part and s2s-large -part.
Table 2 (top) shows that s2s-large performs better than s2s-base and training on more data leads to better performance, in terms of both EM and span EM.
We test three kinds of model updates: s2s-base -part ) s2s-base , s2s-large -part ) s2s-large , and s2s-base ) s2s-large .
Results As shown in Table 2 (bottom), model update regression is prevalent across different model update settings.
The most severe model update regression comes from different pre-trained language models ( s2s-base ) s2s-large ), accounts for 25 .
45% (EM) and 29 .
39% (span EM) errors made by the new model, while the absolute performance gains are merely 1 .
3% EM points and 0 .
77% span EM points.
4.4 Discussions
The key ﬁndings observed across all studied tasks and model updates are the following: (1) Model update regression is prevalent on two typical structured prediction tasks and various model update settings.
(2) Even small changes in training conﬁguration such as changing the random seed can introduce signiﬁcant regression (a minimum of 1.55% NFR is observed).
(3) Heterogeneous model updates lead to severer regression compared to homogeneous model updates.
(4) Updating pre-trained language models leads to higher regressions than those caused by training on more data.
As much as 3 out of 10 test errors are regression errors (NFI ⇡ 30%).
(5) Improving accuracy alone does not necessarily reduce regression (in fact, NFR can be larger than accuracy gain).
First, we describe the adaptation of the methods explored in reducing model update regression in classiﬁcation problems (Shen et al., 2020; Yan et al., 2021; Xie et al., 2021).
Then, we introduce a novel method, backward-congruent re-ranking
that is better suited to structured prediction tasks.
5.1 Model Ensemble
In classiﬁcation problems, (Yan et al., 2021; Xie et al., 2021) found that model ensemble can reduce model update regression without any information about the old model.
This can be attributed to the reduction of variance by ensembling:
Every single model may capture the training data from a distinct aspect.
The ensemble aggregates different aspects and has a shorter average distance to other single models.
Concretely, we train different models with different random seeds and average the local prediction scores from all the models during decoding.
Nevertheless, ensembles may not be practical due to the high cost of training and inference.
We note that parameter averaging (i.e., averaging the parameters of different models and using the resulting model for prediction) is an alternative to prediction ensembling.
However, our preliminary experiments show that this method gives worse performance than prediction ensembling.
5.2 Knowledge Distillation
Knowledge distillation (KD) is a technique originally proposed for model compression (Buciluˇa et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015), where a (smaller) student model is trained to mimic a (larger) teacher model.
By treating the old model as the teacher and the new model as the student,
KD has been proved a promising approach for reducing model update regression in both vision and language classiﬁcation tasks (Yan et al., 2021; Xie et al., 2021).
The instantiations are usually based on a loss function computing the cross-entropy between the output distributions predicted by the teacher and the student.
However, for structured prediction problems, the output space is exponential in size, making the exact output distribution computationally intractable.
Although some efﬁcient methods for approximating the cross-entropy have been proposed (Zmigrod et al., 2021a; Wang et al., 2021), they assume the models follow certain factorizations and thus have limited application scenarios (e.g., the techniques proposed in (Zmigrod et al., 2021a) can only be used for edge-factored parsers).
One may think of performing KD on local predictions or hidden representations instead.
However, such distillation can also be infeasible between models with different factorizations or structures (e.g., deepbiaf ) stackptr in dependency parsing).
To tackle the above problem, we borrow the idea from sequence-level KD (Kim and Rush, 2016), which approximates the teacher distribution with its mode.
Speciﬁcally, sequence-level KD suggests to (1) run the teacher model over the training set to create pseudo training data, (2) then train the student model on this new dataset.
The idea is well-suited to the problem of model update regression because it is model-agnostic; it does not assume any speciﬁc factorizations for the old and new models.
5.3 Backward-Congruent Re-ranking
Unlike model ensemble, knowledge distillation attempts to explicitly align the behaviors of the new model to the old model during training .
Alternatively, we propose Backward-Congruent
Re-ranking (BCR), which does not impose any constraint on the training of the new model and only takes effect during inference .
Re-ranking is a popular approach in structured prediction to combine the strengths of two different models (Collins and Koo, 2005; Socher et al., 2013; Le and Zuidema, 2014; Do and Rehbein, 2020).
It suggests to use one model as the candidate generator for creating a candidate pool and use the other model as the re-ranker for picking the best candidate.
While previous work has been focused on developing powerful re-rankers for overall performance improvement, the purpose of BCR is to reduce model update regression.
BCR treats the new model as the candidate generator and the old model as the re-ranker.
Our motivations are two-fold.
First, structured prediction models can produce many possible predictions.
Second, different predictions may achieve similar error rates but differ by the mistakes they made.
Among them, the most likely one according to the old model should have the least prediction ﬂips.
These make re-ranking particularly useful for structured prediction.
Formally, we have the old model and the new model parameterized by  new and  old respectively.
For a given input x , the new model ﬁrst generates a set of candidates GEN  new ( x ) .
Then we choose the prediction y ⇤ with the highest score computed by the old model.
[Equation]
where p  old ( y | x ) is the old model’s generation probability (score) of y given the input x and GEN can be implemented by various decoding methods, including maximization-based search such as beam search and k -best MST algorithm (Zmigrod et al., 2020, 2021b), and stochastic sampling such as top- k sampling (Fan et al., 2018; Radford et al., 2019) and top- p sampling (Holtzman et al., 2019).
Dropout- p
Sampling Maximization-based search attempts to ﬁnd the top outputs that received the highest scores from the model.
This inherently leads to a set of similar candidates and re-ranking may suffer from the lack of diversity.
On the other hand, stochastic sampling introduces more variances among candidates by randomized choices during decoding.
However, traditional stochastic sampling may suffer from the low quality of sampled outputs due to the deterministic nature of structured prediction.
Furthermore, it is unclear how to adapt popular sampling methods, such as top- k and top- p sampling, to solutions that are not formalized as sequence generation ( e.g. , the biafﬁne parser).
In this work, we explore a special sampling method, dropout- p sampling, that has attracted little attention in the literature.
Dropout (Srivastava et al., 2014) is a regularization technique used in almost all modern neural network-based models.
The key idea is to randomly drop some neurons from the neural network during training.
Normally, dropout is turned off during inference.
However, in dropout- p sampling, we keep using dropout with dropout rate p .
Compared to traditional sampling methods, dropout- p sampling has the following advantages: (1) It directly changes the scoring function and keeps any default decoding algorithm unchanged; (2) It can be regarded as conducting global sampling instead of a series of local sampling at each decoding step, potentially improving the formality of output structure; (3) It also has a broader applicable scope that is not limited to sequence generation models.
Discussion Following previous work (Shen et al., 2004; Yan et al., 2021; Xie et al., 2021), our discussion has been focused on handling one model update.
However, BCR can be extended to handle multiple turns of model updates.
To do so, one can keep the most recent k models and use a weighted combination of their scores as the re-ranking metric.
In practice, k can be set to trade-off between performance and runtime cost.
One downside of BCR is that we must maintain and deploy both the old and new models.
Because the re-ranking step has less time complexity compared to the decoding algorithms and the computation is fully parallelizable, this does not create much additional inference latency.
However, the increase in memory footprint does entail an increase in the inference hosting cost.
One remedy could be to use knowledge distillation to distill the old model(s) into a smaller one, which we leave for future work.
Experiments 6.1 Setup
For the ensembles of biafﬁne parsers, we ﬁrst average the edge scores from each model, then run
MST on the average scores.
For stack-pointer and seq2seq parsers, we average the local prediction scores from each model at each decoding step.
We combine 5 models in model ensemble.
The impact of ensemble size is discussed in Appendix B. For knowledge distillation, as mentioned in Section 5.2, we simply replace the ground-truth output in the original training set with the old model’s predictions when training the new model.
For BCR, various decoding methods are explored for candidate generation.
Speciﬁcally, we use k -best spanning trees algorithm
(Zmigrod et al., 2020, 2021b) with biafﬁne parsers, and beam search with stack-pointer parsers and seq2seq parsers.
We also explore
[Table]
sampling-based decoding methods such as top- k sampling ( k 2 { 5 , 10 , 50 , 100 } )
(Fan et al., 2018;
Radford et al., 2019), top- p sampling ( p 2 { 0 .
(Holtzman et al., 2019), and dropout- p sampling ( p 2 { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } ).
The number of candidates in BCR is set to 10 .
Our experiments show that the best results are always achieved using dropout- p sampling.
For brevity, we only report the best results and defer the detailed analysis of the candidate generation methods to Section 6.4.
We include the model update regression results without any treatment in Section(denoted as Untreated) for reference (i.e., the new models are trained normally with a different set of random seeds).
For dependency parsing, we adapt the implementation of the biafﬁne parser Dozat and Manning (2017) and the stack-pointer parser Ma et al.
(2018) in NeuroNLP2 6 .
We use the default conﬁgurations suggested in (Ma et al., 2018) for training and testing models.
For conversational semantic parsing, we re-implement the seq2seq parser Rongali et al.
(2020) using Fairseq Ott et al.
and the pre-trained language models Liu et al.
(2019) are downloaded from HuggingFace 7 .
6.2 Results on Dependency Parsing
Table 3 shows the results of different methods for reducing model update regression on the dependency parsing task.
We can see that model ensemble brings down negative ﬂips in all model update settings compared with the untreated baseline (on average 28% relative reduction in NFI).
On the other hand, knowledge distillation shows little effect on reducing model update regression.
We hypothesize that it is due to the almost perfect word-level accuracy (UAS) of the old model when running on the training set.
As a result, there is little difference between the original training set and the synthetic training set used for distillation, leading to performance close to the untreated baseline.
For BCR, it substantially reduces NFR and NFI across all model update settings (an average of 58% relative reduction in NFI), greatly outperforming model ensemble and knowledge distillation.
This is very encouraging given that model ensemble is often the most effective approach to reduce regression in classiﬁcation problems (Yan et al., 2021; Xie et al., 2021) or even considered as a paragon (Yan et al., 2021).
Interestingly, BCR delivers more pronounced gains over model ensemble in the heterogeneous model update setting ( deepbiaf ) stackptr ).
In particular, BCR reduces NFI by 67% while model ensemble only obtains 21% relative reduction, according to UCM.
The reason might be that model ensemble can only reduce the variance in models of the same kind, while the intrinsic gap between different kinds of models cannot be reduced.
In contrast, BCR selects the most backward-compatible prediction regardless of the source of candidates.
Another interesting observation is that BCR can also improve the overall accuracies though the improvements slightly lag behind model ensemble (e.g., +0 . 19 vs. +0 .
37 UAS points on average).
We regard it as a side beneﬁt since our primary goal here is to reduce model update regression rather than to boost accuracy.
6.3 Results on Conversational Semantic Parsing
Table 4 shows the results of different methods for reducing model update regression on the conversational semantic parsing task.
The empirical results generally support the main ﬁndings on the
[Table] [Table]
dependency parsing task: (1) Knowledge distillation has little effect on reducing NFR and NFI (except for the s2s-base ) s2s-large setting).
(2) Model ensemble brings moderate reduction ( 28% relative
NFI reduction on average) across all model update settings.
(3) BCR achieves substantial reduction ( 73% relative NFI reduction on average), greatly outperforming model ensemble and knowledge distillation.
BCR not only reduces model update regression but also improves accuracies as compared to the untreated baseline.
This is particularly intriguing since we are using a weaker model (the old model) is used to re-rank the candidates generated by a stronger model (the new model).
We speculate that the reason is the complementary effect of different models in differentiating distractors.
6.4 Analysis of BCR
Next, we study the two key components of BCR, namely candidate generation and the re-ranking metric.
We analyze the results of different decoding methods described in Section 5.3 and empirically reveal the performance upper-bounds of BCR.
Candidates for Re-ranking We present the results of BCR with different decoding methods in
Table 5.
For clarity, we only report the best result obtained using each decoding method together with the optimal hyper-parameter setting (except for dropout- p sampling).
We can see that dropout- p sampling obtains the largest reduction in both NFR and NFI, substantially outperforming beam search and traditional truncated sampling (the gaps in NFI are larger than 10 points).
top- p and top- k sampling are the least effective approaches.
We hypothesize that traditional truncated sampling is not suitable for our structured prediction tasks.
Unlike open-ended generation tasks, there are only a few valid local predictions at each decoding step.
Therefore, the candidates generated by truncated sampling may lack diversity.
For dropout- p sampling, as p increases from 0 .
1 to 0 .
3 , the improvements in accuracies shrink while the reductions in NFR and NFI increase.
This is expected since a larger p leads to lower quality of individual candidates but a more diverse candidate pool.
We plot the curves of NFR, NFI, and error rate (ER) with the number of candidates in Figure 2.
As seen, all three metrics decrease as the number of candidates increases.
However, they converge at different points.
NFI and NFR drop signiﬁcantly even after the reduction in ER reaches its limit.
Oracle Re-ranker To shed light on the effectiveness of our choice of the re-ranking metric (i.e., the old model’s prediction score p  old ), we compare it to two kinds of oracle re-rankers: one using span-level EM (w/ ACC) and one using span-level NFR (w/ NFR).
The ACC and NFR re-rankers represent the upper bounds of ACC and NFR that re-ranking methods can achieve.
The results are presented in Table 6.
It can be observed that the NFI and NFR of using p  old are very close to the upper bounds for reducing model update regression (w/ NFR).
Another notable observation is that the NFIs of using p  old and NFR re-ranking are much lower than the NFI of ACC re-ranking.
This demonstrates that improving accuracy is not equivalent to reducing model update regression.
[Table]
Inference Speed For dropout- p sampling, the overall computation overhead of the decoding step grows linearly with the number of candidates.
Nevertheless, different runs of sampling can be done in parallel.
With the same inference hardware (one Nvidia V
GPU) and the same batch size of 32, the decoding and re-ranking speeds of deepbiaf are 171 andsentences per second, and 64 and 221 sentences per second for stackptr .
For seq2seq parsers, we observe that the speed of re-ranking is about 5x of the decoding speed.
In practice, the re-ranking step can be made even faster as it generally allows for larger batch sizes.
Conclusions
This paper presented the ﬁrst study on the model update regression issue in NLP structured prediction tasks.
Experiments on two representative structured prediction tasks showed that model update regression is severe and widespread across different model update settings.
To reduce model update regression, we explored knowledge distillation, model ensemble, and backward-congruent re-ranking.
Backward-congruent re-ranking consistently achieves much more signiﬁcant regression reduction than distillation and ensemble methods.
In the future, model update regression should be examined in other structured prediction tasks such as text generation in NLP and image segmentation in computer vision, and scenarios of multiple rounds of model updates.