Larger-Context Tagging: When and Why Does It Work? (2021)

Abstract: The development of neural networks and pretraining techniques has spawned many sentence-level tagging systems that achieved superior performance on typical benchmarks. However, a relatively less discussed topic is what if more context information is introduced into current top-scoring tagging systems. Although several existing works have attempted to shift tagging systems from sentence-level to document-level, there is still no consensus conclusion about when and why it works, which limits the applicability of the larger-context approach in tagging tasks. In this paper, instead of pursuing a state-of-the-art tagging system by architectural exploration, we focus on investigating when and why the larger-context training, as a general strategy, can work. To this end, we conduct a thorough comparative study on four proposed aggregators for context information collecting and present an attribute-aided evaluation method to interpret the improvement brought by larger-context training. Experimentally, we set up a testbed based on four tagging tasks and thirteen datasets. Hopefully, our preliminary observations can deepen the understanding of larger-context training and enlighten more follow-up works on the use of contextual information.

## introduction the rapid development of deep neural models has shown impressive performances on sequence tagging tasks that aim to assign labels to each token of an input sequence @xcite @xcite . more recently, the use of unsupervised pre-trained models @xcite @xcite (especially contextualized version) has driven state-of-the-art performance to a new level. among these works, researchers frequently choose the boundary with the granularity of sentences for tagging tasks (i.e., sentence-level tagging) @xcite @xcite . undoubtedly, as a transient, sentence-level setting enables us to develop numerous successful tagging systems, nevertheless the task itself should have not be defined as sentence-level but for simplifying the learning process for machine learning models. naturally, it would be interesting to see what if larger-context information (e.g., taking information of neighbor sentences into account) is introduced to modern top-scoring systems, which have shown superior performance under the sentencelevel setting. a small number of works have made seminal exploration in this direction, in which part of works show significant improvement of largercontext @xcite while others don't @xcite @xcite . therefore, it's still unclear when and why largercontext training is beneficial for tagging tasks. in this paper, we try to figure it out by asking the following three research questions: q1: how do different integration ways of largercontext information influence the system's performance? the rapid development of neural networks provides us with diverse flavors of neural components to aggregate larger-context information, which, for example, can be structured as a sequential topology by recurrent neural networks @xcite (rnns) or graph topology by graph neural networks @xcite . understanding the discrepancies of these aggregators can help us reach a more generalized conclusion about the effectiveness of larger-context training. to this end, we study larger-context aggregators with three different structural priors (defined in sec. 3.2) and comprehensively evaluate their efficacy. q2: can the larger-context training easily play to its strengths with the help of recently arising contextualized pre-trained models @xcite @xcite (e.g. bert)? the contextual modeling power of these pre-trained methods makes it worth looking at its effect on larger-context training. in this work, we take bert as a case study and assess its effectiveness quantitatively and qualitatively. q3: if improvements could be observed, where does the gain come and how do different characteristics of datasets affect the amount of gain? instead of simply figuring out whether larger-context training could work, we also try to interpret its gains. specifically, we propose to use fine-grained evaluation to explain where the improvement comes from and why different datasets exhibit discrepant gains. overall, the first two questions aim to explore when larger-context training can work while the third question addresses why. experimentally, we try to answer these questions by conducting a comprehensive analysis, which involves four tagging tasks and thirteen datasets. our main observations are summarized in sec. 8. 1 furthermore, we show, with the help of these observations, it's easier to adapt larger-context training to modern top-performing tagging systems with significant gains. we brief our contributions below: 1) we try to bridge the gap by asking three research questions, between the increasing topperforming sentence-level tagging systems and insufficient understanding of larger-context training, encouraging future research to explore more largercontext tagging systems. 2) we systematically investigate four aggregators for larger-context and present an attribute-aided evaluation methodology to interpret the relative advantages of them, and why they can work (sec. 3.2). 3) based on some of our observations, we adapt larger-context training to five modern top-scoring systems in the ner task and observe that all larger-context enhanced models can achieve significant improvement (sec. 6). encouragingly , with the help of larger-context training, the performance of @xcite on the wb (ontonotes5.0-wb) dataset can be improved by a 10.78 f 1 score. 1 putting the conclusion at the end can help the reader understand it better since more contextual information about experiments has been introduced. ## task, dataset, and model we first explicate the definition of tagging task and then describe several popular datasets as well as typical methods of this task. ## experiment: why does it work? experiments in this section are designed for the research questions q3, interpreting where the gains of a larger-context approach come and why different datasets exhibit diverse improvements. to achieve this goal, we use the concept of interpretable evaluation @xcite ) that allows us perform fine-grained evaluation of one or multiple systems. ## adapting to top-scoring systems beyond the above quantitative and qualitative analysis of our instantiated typical tagging models (sec.2.3), we are also curious about how well modern top-scoring tagging systems perform when equipped with larger-context training. to this end, we choose the ner task as a case study and first re-implement existing topperforming models for different ner datasets separately, and then adapt larger-context approach to them based on the seq or cpre-seq aggregator, 7 which has shown superior performance in our above analysis. settings we collect five top-scoring tagging systems @xcite @xcite @xcite ) that 7 training all four aggregators for all tagging tasks is much more costly and here we choose these two since they can obtain better performance at a relatively lower cost. are most recently proposed foot_5 . among these five models, regarding @xcite , we use cpreseq aggregator for the larger-context training, since this model originally relies on a contextualized pre-trained layer. besides, from above analysis in sec. 5.4 we know the suitable datasets for cpre-seq aggregator: cn03, wb, tc, bc, and nw. regarding the other four models, we use the seq aggregator for the larger-context training and the matched datasets are: wb, tc, and bc. results tab. 3 shows the relative improvement of larger-context training on five modern top-scoring models in the ner task. we observe that the largercontext training has achieved consistent gains on all chosen datasets, which holds for both seq and cpre-seq aggregators. notably, the larger-cotext training achieves sharp improvement on wb, which holds for all the five top-scoring models. for example, with the help of larger-context training, the performance can be improved significantly using @xcite and 7.18 f 1 score using @xcite . this suggests that modern top-scoring ner systems can also benefit from larger-context training. ## related work our work touches the following research topics for tagging tasks. sentence-level tagging existing works have achieved impressive performance at sentence-level tagging by extensive structural explorations with different types of neural components. regarding sentence encoders, recurrent neural nets @xcite @xcite hovy, models aggregator datasets norm seq cpre bc wb tc cn03 nw luo et al. (2020) √ 78.78 62.38 65.56 --√ +0.54 +7.18 +1.7 lin et al. (2019) √ 77.80 63.16 65.19 --√ +2.94 +5.07 +2.25 chen et al. (2019) √ 77.50 66.51 65.49 --√ +1.96 +3.98 +0.89 --yan et al. (2019) √ 81.29 65.05 67.92 92.17 90.37 √ +0.16 +6.79 +3.03 +0.04 +1.11 akbik et al. (2018) √ 81.13 64.79 69.00 93.03 90.76 √ +1.12 +10.78 +2.12 +0.05 +1.03 document-level tagging document-level tagging introduced more contextual features to improve the performance of tagging. some early works introduced non-local information @xcite to enhance traditional machine learning methods (e.g., crf @xcite ) and achieved impressive results. qian et al. (2018); wadden et al. (2019) built graph representation based on the broad dependencies between words and sentences. @xcite proposed to use a memory network to record the document-aware information. besides, document-level features was introduced by different domains to alleviate label inconsistency problems, such as news ner @xcite , chemical ner @xcite , disease ner @xcite , and chinese patent @xcite . compared with these works, instead of proposing a novel model, we focus on investigating when and why the larger-context training, as a general strategy, can work. interpretability and robustness of sequence labeling systems recently, there is a popular trend that aims to (i) perform a glass-box analysis of sequence labeling systems @xcite , understanding their generalization ability and quantify robustness @xcite , (ii) interpretable evaluation of them @xcite , making it possible to know what a system is good/bad at and where a system outperforms another, (iii) reliable analysis @xcite for test set with fewer samples. our work is based on the technique of interpretable evaluation, which provides a convenient way for us to diagnose different systems. ## discussion we summarize the main observations from our experiments and try to provide preliminary answers to our proposed research questions: (i) how do different integration ways of largercontext information influence the system's performance? overall, introducing larger-context information will bring gains regardless of the ways how to introduce it (e.g., seq, graph). particularly, larger-context training with seq aggregator can achieve better performance at lower training cost compared with graph and bow aggregators (sec. 4.1).