Enhanced Language Representation with Label Knowledge for Span Extraction (2021)

Abstract: Span extraction, aiming to extract text spans (such as words or phrases) from plain text, is a fundamental process in Information Extraction. Recent works introduce the label knowledge to enhance the text representation by formalizing the span extraction task into a question answering problem (QA Formalization), which achieves state-of-the-art performance. However, such a QA Formalization does not fully exploit the label knowledge and causes a dramatic decrease in efficiency of training/inference. To address those problems, we introduce a fresh paradigm to integrate label knowledge and further propose a novel model to explicitly and efficiently integrate label knowledge into text representations. Specifically, it encodes texts and label annotations independently and then integrates label knowledge into text representation with an elaborate-designed semantics fusion module. We conduct extensive experiments on three typical span extraction tasks: flat NER, nested NER, and event detection. The empirical results show that 1) our model achieves a new state-of-the-art performance on four benchmarks, and 2) reduces training time and inference time by 76% and 77% on average, respectively, compared with the QA Formalization paradigm.

## introduction information extraction (ie), a fundamental task in natural language processing, aims to extract structured knowledge from unstructured texts. it usually contains the process that extracts text spans (such as words or phrases) from plain text, e.g., ner. span extraction is usually formulated into the sequence labeling problem that assigns a categorical label to each token in a text. many efforts have been devoted to span extraction. early approaches are mainly based on handcrafted features such as domain dictionaries @xcite and lexical features @xcite . as neural networks show the effectiveness of learning text features automatically, many neural-based methods have been proposed @xcite @xcite . recently, self-attention-based pre-trained language models such as bert @xcite are widely used to boost the span extraction task @xcite . however, most existing methods treat labels as independent and meaningless one-hot vectors, neglecting prior information of labels (referred to as label knowl-edge). figure 2 : the visualization of attention mechanism for the token "judge" (qa formalization). 2 the darker color indicates the higher attention score. to alleviate the limitation, several studies @xcite @xcite start to integrate label knowledge into span extraction. among them, qa formalization is especially attractive due to its effectiveness @xcite @xcite @xcite . simply put, qa formalization treats span extraction as a question answering problem. taking ner as an example, to extract "person" entities, it is formalized as answering the question "which person is mentioned in the text, in which a person represents a human or individual?" based on the given text. benefiting from the label knowledge of the category-related questions, qa formalization usually yields stateof-the-art performance in span extraction even in low-resource scenarios. qa formalization, however, exhibits two key weaknesses: 1) inefficiency: formalizing the span extraction as qa causes a drastic reduction of training/inference efficiency. specifically, the typical qa formalization method concatenates question and text as the input (e.g., [cls] question [sep] text [sep]) and jointly encodes question and text with a transformer-based encoder. the jointencoding has to transform every text into |c| pairs of the form question, text , where |c| is the size of the label category set. this transformation, which increases both the size of the sample set and the length of text sequences, finally increases the time cost of training and inference. 2) underutilization: the label knowledge is integrated implicitly into text representation based on the self-attention mechanism @xcite . as figure 2 shows, the "attention" of self-attention mechanism will be distracted by text, not entirely focus on the question part. thus, the label knowledge is not fully exploited to enhance the text representations. to address aforementioned two problems, we propose a novel paradigm (seen in figure 1 ) to integrate label knowledge. first, since joint-encoding causes low efficiency, we decompose question-text encoding process into two separate encoding modules: the text encoding module f 1 and the question encoding module f 1 . in this way, the size of the sample set is no longer expanded by |c| times. second, to fully utilize the label knowledge, a fusion module f is designed to explicitly integrate the label and the text representations. to instantiates the above paradigm, we further propose a model termed as lear to learn labelknowledge enhanced representation. a powerful encoder f 1 is essential for understanding the label annotations foot_2 . however, training the encoder f 1 from scratch is challenging since the number of label annotations is too small. thus we share the weights of f 1 and f 1 (called shared encoder), which can learn the label knowledge by large pretrained model and does not introduce extra parameters. next, the learned label knowledge is integrated into text representations by the semanticsguided attention module. we conduct experiments in five benchmarks on three typical span extraction tasks: flat ner, nested ner, and event detection (ed). compared with qa formalization baselines, our model lear outperforms them to achieve a new state-of-the-art. furthermore, lear reduces training time and inference time by 76% and 77% on average, respectively. to sum up, our contributions are as follows: ## approach in this section, we first give an overall description of our lear architecture. lear consists of three crucial modules: semantics encoding module, semantics fusion module, and span decoding module. our architecture (figure 3 ) takes text x and label annotation y of category set c as input. the two inputs are respectively processed by two encoder networks whose backbone is bert @xcite . the two encoders share weights (referred to as shared encoder) while processing the two inputs. then the text embedding and label embedding produced by the shared encoder are fused by the semantic fusion module to derive the labelknowledge-enhanced embeddings for the text. finally, the label-knowledge-enhanced embeddings are used to predict whether or not each token is a start or end index for some category. ## experiments in this section, we present lear results on 5 widely-used benchmarks. ## related work event detection (ed). event detection aims at extracting event triggers from a text and classifying them. it is dominantly solved in a representationbased manner, where triggers are represented by embedding. in case of no extra information, the representation can be obtained by a powerful text encoder which is usually based on cnn @xcite , @xcite , or attention mechanism @xcite . besides, the representation can be enhanced by extra information. examples of typical extra information include syntactic information @xcite and knowledge base @xcite . in particular, label knowledge is attracting more and more attention @xcite , which usually formalizes ed as a qa problem. named entity recognition (ner). named entity recognition seeks to locate named entities in an unstructured text and classify them into pre-defined categories such as person, organization, location, etc. traditional methods treat it as a classification task and use crfs @xcite as the backbone. then neural networks become a prevalent tool in ner with the development of deep learning. recently, the performance of ner has been further improved by large-scale language models such as elmo @xcite and bert @xcite . when label knowledge is available, state-of-the-art performance can be obtained by formulating ner as a qa problem. ## conclusion in this paper, we propose a novel paradigm to exploit label knowledge to boost the span extraction task and further instantiate a model named lear. unlike the existing qa formalization methods, lear first encodes the text and label annotations independently, and uses a semantic fusion module to integrate label knowledge into the text representation explicitly. in this way, we can overcome the