Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers (2021)

Abstract: We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.

## introduction sequence labeling and sentence classification can represent facets of the same task at different granularities; for example, detecting grammar errors and predicting the grammaticality of sentences. transformer-based architectures such as bert @xcite and roberta @xcite have been shown to achieve state-of-the-art results on both sequence labeling @xcite and sentence classification @xcite problems. however, such tasks are typically treated in isolation rather than within a unified approach. in this paper, we investigate methods for inferring token-level predictions from transformer models trained only on sentence-level annotations. the ability to classify individual tokens without direct supervision opens possibilities for training sequence labeling models on tasks and datasets where only sentence-level or document-level annotation is available. in addition, attention-based architectures allow us to directly investigate what the model is learning and to quantitatively measure whether its rationales (supporting evidence) for particular input sentences match human expectations. while evaluating the faithfulness @xcite of a model's rationale is still an open research question and up for debate @xcite @xcite @xcite , the methods explored here allow for measuring the plausibility (agreeability to human annotators; deyoung et al. ( 2020 )) of transformer-based models using existing sequence labeling datasets. we evaluate and compare different methods for adapting pre-trained transformer models into zeroshot sequence labelers, trained using only gold sentence-level signal. our experiments show that applying existing approaches @xcite to transformer architectures is not straightforward -transformers already contain several layers of multi-head attention, distributing sentencelevel information across many tokens, whereas the existing methods rely on all the information going through one central attention module. approaches such as lime @xcite for scoring word importance also struggle to infer correct token-level annotations in a zero-shot manner (e.g., it achieves only 2% f-score on one of our datasets). we find that a modified attention function is needed to allow transformers to better focus on individual important tokens and achieve a new state-of-the-art on zero-shot sequence labeling. the contributions of this paper are fourfold: ## methods we evaluate four different methods for turning sentence-level transformer models into zero-shot sequence labelers. ## datasets we investigate the performance of these methods as zero-shot sequence labelers using three different datasets. gold token-level annotation in these datasets is used for evaluation; however, the models are trained using sentence-level labels only. the conll 2010 shared task @xcite foot_1 focuses on the detection of uncertainty cues in natural language text. the dataset contains 19, 542 examples with both sentence-level uncertainty labels and annotated keywords indicating uncertainty. we use the train/test data from the task and randomly choose 10% of the training set for development. we also evaluate on the task of grammatical error detection (ged) -identifying which sentences are grammatically incorrect (i.e., contain at least one grammatical error). the first certificate in english dataset fce @xcite consists of essays written by non-native learners of english, annotated for grammatical errors. we use the train/dev/test splits released by @xcite for sequence labeling, with a total of 33, 673 sentences. in addition, we evaluate on the write & improve @xcite and locness @xcite ged dataset foot_2 (38, 692 sentences) released as part of the bea 2019 shared task @xcite . it contains english essays written in response to varied topics and by english learners from different proficiency levels, as well as native english speakers. as the gold test set labels are not publicly available, we evaluate on the released development set and use 10% of the training data for tuning foot_3 . for both ged datasets, we train the model to detect grammatically incorrect sentences and evaluate how well the methods can identify individual tokens that have been annotated as errors. ## experimental setup we use the pre-trained roberta-base @xcite model, made available by huggingface @xcite , as our transformer architecture. following @xcite , transformer models are fine-tuned for 20 epochs, and the best performing checkpoint is then chosen based on sentence-level performance on the development set. each experiment is repeated with 5 different random seeds and the averaged results are reported. the average duration of training on nvidia geforce rtx 2080ti was 1 hour. significance testing is performed with a two-tailed paired t-test and @xmath0.05. hyperparameteres are tuned on the development set and presented in appendices b and c. the lime and attention head methods provide only a score without a natural decision boundary for classification. therefore, we choose their thresholds based on the token-level f 1 -score on the development set. in contrast, the soft attention and weighted soft attention methods do not require such additional tuning that uses token-level labels. ## results the results are presented in we report the f 1 -measure on the token level along with mean average precision (map) for returning positive tokens. the map metric views the task as a ranking problem and therefore removes figure 1 : example word-level importance scores r i (eq. 6) of different methods applied to an excerpt from the conll10 dataset. head corresponds to attention heads; sa to soft attention; and w-sa to weighted soft attention. we can observe how w-sa is the only method that correctly assigns substantially higher weights to the 'may' and 'seems' uncertainty cues. the dependence on specific classification thresholds. in addition, we report the f 1 -measure on the main sentence-level task to ensure the proposed methods do not have adverse effects on sentence classification performance. precision and recall values are included in appendix e. lime has relatively low performance on fce and bea 2019, while it achieves somewhat higher results on conll 2010. comparing the map scores, the attention head method performs substantially better, especially considering that it is much more lightweight and requires no additional computation. nevertheless, both of these methods rely on using some annotated examples to tune their classification threshold, which precludes their application in a truly zero-shot setting. combining the soft attention mechanism with the transformer architecture provides some improvements over the previous methods, while also improving over @xcite . a notable exception is the conll 2010 dataset where this method achieves only 8% f 1 and 20% map. error analysis revealed that this is due to the transformer representations spreading attention scores evenly between a large number of tokens, as observed in figure 1 . uncertainty cues in conll 2010 can span across whole sentences (e.g., 'either ... or ...'), with such examples encouraging the model to distribute information even further. the weighted soft attention modification addresses this issue and considerably improves performance across all metrics on all datasets. compared to the non-weighted version of the soft attention method, applying the extra weights leads to a significant improvement in terms of map, with a minimum of 5.01% absolute gain on fce. the improvements are also statistically significant compared to the current state of the art @xcite : 5.35% absolute improvement on fce; 9.38% on bea 2019; and 3.36% on conll 2010. while the f 1 on conll 2010 is slightly lower, the map score is higher, indicating that the model has difficulty finding an optimal decision boundary, but nevertheless provides a better ranking. in future work, the weighted soft attention method for transformers could potentially be combined with token supervision in order to train robust multi-level models @xcite . ## conclusion we investigated methods for inferring tokenlevel predictions from transformer models trained only on sentence-level annotations. experiments showed that previous approaches designed for lstm architectures do not perform as well when applied to transformers. as transformer models already contain multiple layers of multi-head attention, the input representations get distributed between many tokens, making it more difficult to identify the importance of each individual token. lime was not able to accurately identify target tokens, while the soft attention method primarily assigned equal attention scores across most words in a sentence. directly using the scores from the existing attention heads performed better than expected, but required some annotated data for tuning the decision threshold. modifying the soft attention module with an explicit sharpness constraint on the weights was found to encourage more distinct predictions, significantly improving token-level results.