Overcoming Conflicting Data when Updating a Neural Semantic Parser (2021)

Abstract: In this paper, we explore how to use a small amount of new data to update a task-oriented semantic parsing model when the desired output for some examples has changed. When making updates in this way, one potential problem that arises is the presence of conflicting data, or out-of-date labels in the original training set. To evaluate the impact of this understudied problem, we propose an experimental setup for simulating changes to a neural semantic parser. We show that the presence of conflicting data greatly hinders learning of an update, then explore several methods to mitigate its effect. Our multi-task and data selection methods lead to large improvements in model accuracy compared to a naive data-mixing strategy, and our best method closes 86% of the accuracy gap between this baseline and an oracle upper bound.

## introduction most work in semantic parsing @xcite considers a scenario where the desired outputs of a model are static and can be specified with a large, fixed dataset. however, when deploying a semantic parsing model in a real world virtual assistant, it is often necessary to update a model to support new features or to enable improvements in downstream processing. for example, creators of a media assistant may want to add new commands specialized towards new media types like podcasts, or those of a navigation assistant might like to add a new feature to allow users to specify roads to avoid. such changes require that the structures output by the model are updated, either by introducing a new intent (a representation of a desired action) or re-configuring arguments of existing intents. in this work, we investigate the best way to make updates to the intents and arguments of a task-oriented neural semantic parsing model. to make an update, new data annotations must be collected to specify the new form that is desired for model outputs. because changes can be quite frequent, we would like to be able to collect a small amount of data for each update (on the order of 50 examples) and merge the new information with a much larger existing dataset. naively, we might hope to simply combine new data in with the old and train on the combination. however, this approach has the problem that some of the older data with out-of-date labels may conflict with the new labeling. these conflicts occur whenever inputs that would be affected by a change appear in the original dataset. for example, when introducing a new intent for podcasts, the original dataset may have included podcast examples labeled with a more generic media intent or a label indicating that the feature is unsupported. when introducing a new argument, say 'roads to avoid', there may be instances in the original dataset that should have this argument labeled but do not because they were annotated before the argument was introduced. this conflicting data can confuse the model and cause it to predict labels as they were before an update rather than after. unfortunately, this problem of conflicting data during model updates is understudied in the academic literature. to enable exploration of this problem on publicly available datasets, we propose a method to easily create simulated updates with conflicts (section 3.3) and release our update data. the idea behind our method is to form updates in the reverse direction, relabeling instances of a particular intent or argument to simulate out-of-date labels. using our proposed setup, we demonstrate how conflicting data greatly hinders learning of updates (section 5). in addition, we explore several methods for mitigating the negative effects of conflicting data by modifying how we combine the old and the new data (section 6). one approach is to keep the old and new datasets separate, but to share information indirectly with either fine-tuning or multi-task learning. another approach is to explicitly filter data that is likely to conflict using a learned classifier. each of these methods substantially improves performance compared to naively mixing in new data, establishing strong baselines for future work on this task. in summary, the contributions of this work are 1) establishing an experimental setup to test updates with conflicting data, 2) demonstrating that conflicting data leads to large losses in model performance if left unmitigated, and 3) exploring several possible approaches to mitigate this problem. ## related work there has been a substantial amount of prior work on making updates to neural models @xcite @xcite @xcite , demonstrating a recognition in the community that the ability to update models is important. however, most of these works consider a setting where none of the original data conflicts with the new labels. thus these works, and the general continual learning and class-incremental learning literature, assume that the space of inputs affected by a change does not appear at all in the original data. in many scenarios, this assumption does not hold because the original dataset will aim to cover the full distribution of inputs a model might encounter. because of the non-conflicting assumption, this body of prior work focuses on other questions such as what can be done when the original data is no longer available. one paper that does consider updates with label conflicts is @xcite . although they do not intentionally set out to study conflictingdata updates, in their ner task locations where new labels apply are tagged with special "outside" labels prior to the update, which cause conflicts with the new labels. while their work avoids the conflicting data problem by considering a setting where the original data is no longer available, our experiments show that it can be advantageous to instead keep the original data around and use more direct methods to avoid problems of conflicting data. our work also has parallels to the concept drift literature @xcite . however, concept drift work focuses on unintentional changes to a natural data distribution over time, while our work is concerned with intentional restructuring of the output representation space, leading to very a different setup and choice of methods. in particular, that work operates over a stream of examples where changes often occur gradually, does not generally include structured prediction tasks, and does not allow the practitioner to introduce additional structure on the types of annotation given (as we do in section 3.2). finally, our work relates to work on training with noisy labels @xcite @xcite , since the incorrect labels from conflicting data could be viewed as a type of noise. however, it is important to evaluate the problem of conflict-causing updates separately from other noisy label problems because the distribution of incorrect labels due to an update will be very different from most other sources of label noise. while not all noisy-label methods can directly apply to our task (as many are designed for classification as opposed to structured prediction) and may not take full advantage of the additional structure of our problem, we believe this line of work can still serve as a source of inspiration for future exploration on our task. ## base model the semantic parsing model we use for our experiments is a sequence-to-sequence model based on the transformer architecture @xcite . we use a sequence-to-sequence model because of their flexibility and widespread use in semantic parsing @xcite @xcite and nlp in general. our model encodes a language input using a pretrained 12-layer bert model @xcite , then decodes a parse tree flattened by depth-first traversal. at each step, the decoder can generate either 1) a labeled bracket representing an intent or argument label 2) a closing bracket or 3) an index of an input token to be copied. the hyperparameters of our model architecture and training can be found in appendix a. ## effect of conflicting data before we describe and test our methods for mitigating conflicts, this section will briefly explore how conflicting data affects learning. we evaluate model updates both with and without conflicting data, and compare accuracies as we vary the amount of new v2 data being introduced. the non-conflicting setting represents an oracle where all conflicting data is removed, which we can easily simulate in our synthetic data-creation process but is generally not achievable on real-world data without additional manual annotation. more precisely, when evaluating updates with conflicting data, we include 50 examples with outof-date labels in the changed category. we mix these examples with a full set of unchanged and trivially-unchanged examples to represent a v1 training set from before an update. we then introduce different amounts of changed examples with updated labels to act as the v2 training set, with sizes ranging from from half the conflicting set size (25) to four times as much @xcite . for each data size, we measure accuracy using an exact match metric, meaning the entire tree output by the model must match the reference to be considered correct. for the non-conflicting setting, we do not include the 50 examples with out-of-date labels, but otherwise use the same setup. the results for this experiment are shown in figure 2 , after averaging over five different changes (a detailed breakdown of results can be found in appendix c). to the left of the graph, we see that when the amount of conflicting data is greater than the size of the new data being added, we get less than half of the accuracy we would get without conflicting data. while the gap narrows somewhat with more data, even when we introduce four times as much new data as there are conflicting examples, the presence of conflicting data still leads to a loss of over 10% accuracy. these results show just how detrimental conflicting data can be to the learning of a model update. ## mitigation methods in this work, we consider three methods to alleviate the problems caused by conflicting data: finetuning, multi-task learning with separate decoder heads, and data filtering with a learned classifier. ## evaluation in this section, we describe the evaluation of our methods for mitigating the conflicting data problem. for each update, we form a test set by randomly selecting 100 examples from each of the three data partitions (changed, unchanged, and triviallyunchanged), and a v2 training set by selecting 50 changed and 50 unchanged examples. all remaining examples are placed into the v1 training set and changed examples are relabeled appropriately. note that unlike in section 5 where we used a fixedsize conflicting set of 50 examples in v1, in this setup we use all remaining examples available after sampling a subset for v2, which results in larger conflicting sets ranging from hundreds to thousands of examples. the large sizes of the conflicting sets further amplifies the effect of the conflicting data. we report results in terms of exact match accuracy between the predicted and target parse structure. results are aggregated across 5 different updates, giving us a total of 1500 test examples for each method (5 updates × 3 partitions × 100 examples). we also average across 5 different runs for each method to reduce variance. we compare against three baselines: training only on the original v1 data, training only on the new v2 data, and directly mixing the two data versions together into a single dataset. 3 we also compare to an upper bound where the entire v1 training set is re-annotated with updated labels. for many updates, this upper bound requires thousands of new annotations, as compared to the one hundred labels used by our methods. ## conclusion this work has shown that in order to make effective updates to the outputs of a neural semantic parsing model by adding new data, it is important to consider the effect of conflicting examples in the original data. conflicting data is likely a problem in many different scenarios where outputs to a model must be updated, and we believe that further study into methods for mitigating the effects of conflicting data is an important direction to allow practitioners to handle the constantly-changing needs of real-world machine learning applications.