Generalizing Natural Language Analysis through Span-relation Representations (2020)

Abstract: Natural language processing covers a wide variety of tasks predicting syntax, semantics, and information content, and usually each type of output is generated with specially designed architectures. In this paper, we provide the simple insight that a great variety of tasks can be represented in a single unified format consisting of labeling spans and relations between spans, thus a single task-independent model can be used across different tasks. We perform extensive experiments to test this insight on 10 disparate tasks spanning dependency parsing (syntax), semantic role labeling (semantics), relation extraction (information content), aspect based sentiment analysis (sentiment), and many others, achieving performance comparable to state-of-the-art specialized models. We further demonstrate benefits of multi-task learning, and also show that the proposed method makes it easy to analyze differences and similarities in how the model handles different tasks. Finally, we convert these datasets into a unified format to build a benchmark, which provides a holistic testbed for evaluating future models for generalized natural language analysis.

## introduction a large number of natural language processing (nlp) tasks exist to analyze various aspects of human language, including syntax (e.g., constituency and dependency parsing), semantics (e.g., semantic role labeling), information content (e.g., named entity recognition and relation extraction), or sentiment (e.g., sentiment analysis). at first glance, these tasks are seemingly very different in both the structure of their output and the variety of information that they try to capture. to handle these different characteristics, researchers usually use specially designed neural network architectures. in this paper we ask the simple questions: are the task-specific architectures really necessary? or with the appropriate representational methodology, can we devise a single model that can performand achieve state-of-the-art performance on -a large number of natural language analysis tasks? interestingly, in the domain of efficient human annotation interfaces, it is already standard to use unified representations for a wide variety of nlp tasks. figure 1 shows one example of the brat @xcite annotation interface, which has been used for annotating data for tasks as broad as part-of-speech tagging, named entity recognition, relation extraction, and many others. notably, this interface has a single unified format that consists of spans (e.g., the span of an entity), labels on the spans (e.g., the variety of entity such as "person" or "location"), and labeled relations between the spans (e.g., "born-in"). these labeled relations can form a tree or a graph structure, expressing the linguistic structure of sentences (e.g., dependency tree). we detail this brat format and how it can be used to represent a wide number of natural language analysis tasks in section 2. the simple hypothesis behind our paper is: if humans can perform natural language analysis in a single unified format, then perhaps machines can as well. fortunately, there already exist nlp models that perform span prediction and prediction of relations between pairs of spans, such as the endto-end coreference model of @xcite . we extend this model with minor architectural modifications (which are not our core contributions) and pre-trained contextualized representations (e.g., information extraction pos parsing srl sentiment ner re coref. openie dep. consti. absa orl different models for different tasks elmo @xcite bert @xcite ) spanbert @xcite single model for different tasks @xcite 2016) swayamdipta et al. (2018) strubell et al. (2018) clark et al. (2018) luan et al. (2018, 2019) dixit and al-onaizan (2019) marasović and frank (2018 while previous work has used similar formalisms to understand the representations learned by pretrained embeddings @xcite , to the best of our knowledge this is the first work that uses such a unified model to actually perform analysis. moreover, we demonstrate that despite the model's simplicity, it can achieve comparable performance with special-purpose state-of-the-art models on the tasks above ( ## span-relation representations in this section, we explain how the brat format can be used to represent a large number of tasks. there are two fundamental types of annotations: span annotations and relation annotations. given a sentence @xmath0, w 2 , ..., w n ] of n tokens, a span annotation (s i , l i ) consists of a contiguous span of tokens s @xmath1, w b i +1 , ..., w e i ] and its label l i (l @xmath2, where b i /e i are the start/end indices respectively, and l is a set of span labels. a relation annotation (s j , s k , r jk ) refers to a relation r jk (r @xmath3, where r is a set of relation types. this span-relation representation can easily express many tasks by defining l and r accordingly, as summarized in task spans annotated with labels ner barack obama person was born in hawaii location . consti. and their suspicions np of each other np pp np run deep advp vp . s pos what wp kind nn of in memory nn ? absa great laptop that offers many great features positive ! detailed explanations of all tasks can be found in appendix a. while the tasks above represent a remarkably broad swath of nlp, it is worth mentioning what we have not covered, to properly scope this work. notably, sentence-level tasks such as text classification and natural language inference are not covered, although they can also be formulated using this span-relation representation by treating the entire sentence as a span. we chose to omit these tasks because they are already well-represented by previous work on generalized architectures @xcite and multi-task learning @xcite , and thus we mainly focus on tasks using phrase-like spans. in addition, the span-relation representations described here are designed for natural language analysis, and cannot handle tasks that require generation of text, such as machine translation @xcite , dialog response generation @xcite , and summarization @xcite . there are also a small number of analysis tasks such as semantic parsing to logical forms @xcite where the outputs are not directly associated with spans in the input, and handling these tasks is beyond the scope of this work. ## span-relation model now that it is clear that a very large number of analysis tasks can be formulated in a single format, we turn to devising a single model that can solve these tasks. we base our model on a span-based model first designed for end-to-end coreference resolution @xcite , which is then adapted for other tasks @xcite @xcite @xcite . at the core of the model is a module to represent each span as a fixed-length vector, which is used to predict labels for spans or span pairs. we first briefly describe the span representation used and proven to be effective in previous works, then highlight some details we introduce to make this model generalize to a wide variety of tasks. span representation given a sentence @xmath4, w 2 , ..., w n ] of n tokens, a span s @xmath5, w b i +1 , ..., w e i ] is represented by concatenating two components: a content representation z c i calculated as the weighted average across all token embeddings in the span, and a boundary representation z u i that concatenates the embeddings at the start and end positions of the span. specifically, ## glad benchmark and results we first describe our general language analysis datasets (glad) benchmark and evaluation metrics, then conduct experiments to (1) verify that spanrel can achieve comparable performance across all tasks (section 4.2), and (2) demonstrate its benefits in multi-task learning (section 4.3). ## conclusion we provide the simple insight that a large number of natural language analysis tasks can be represented in a single format consisting of spans and relations between spans. as a result, these tasks can be solved in a single modeling framework that first extracts spans and predicts their labels, then predicts relations between spans. we attempted 10 tasks with this spanrel model and show that this generic task-independent model can achieve competitive performance as state-of-the-art methods tailored for each tasks. we merge 8 datasets into our glad benchmark for evaluating future models for natural language analysis. future directions include (1) devising hierarchical span representations that can handle spans of different length and diverse content more effectively and efficiently; (2) robust multitask learning or meta-learning algorithms that can reconcile very different tasks. hu xu, bing liu, lei shu, and philip yu. 2019. bert post-training for review reading comprehension and aspect-based sentiment analysis. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 2324-2335, minneapolis, minnesota. association for computational linguistics. bishan yang and claire cardie. 2013. joint inference for fine-grained opinion extraction. in proceedings of the 51st annual meeting of the association for computational linguistics, acl 2013, 4-9 august 2013, sofia, bulgaria, volume 1: long papers, pages 1640-1649. junlang zhang and hai zhao. 2019. span based open information extraction. corr, abs/1901.10879.