A Cross-Task Analysis of Text Span Representations (2020)

Abstract: Many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution. While extensive research has focused on functional architectures for representing words and sentences, there is less work on representing arbitrary spans of text within sentences. In this paper, we conduct a comprehensive empirical evaluation of six span representation methods using eight pretrained language representation models across six tasks, including two tasks that we introduce. We find that, although some simple span representations are fairly reliable across tasks, in general the optimal span representation varies by task, and can also vary within different facets of individual tasks. We also find that the choice of span representation has a bigger impact with a fixed pretrained encoder than with a fine-tuned encoder.

## introduction fixed-dimensional span representations are often used as a component in recent models for a number of natural language processing (nlp) tasks, such as question answering @xcite , coreference resolution @xcite , and constituency parsing @xcite @xcite . such models initialized with contextualized word embeddings @xcite have achieved new state-of-the-art results for these tasks @xcite . since spans can have arbitrary length (i.e., number of tokens), fixed-dimensional span representations involve some form of (parameterized) pooling of the token representations. existing models typically pick a span representation method (dashed boxes in figure 1 ) that works well for the task(s) of interest. however, a comprehensive evaluation com- we follow the "edge probing" setup of @xcite and introduce two new tasks to this setup, namely constituent detection and mention detection, which complement the constituent labeling and coreference arc prediction tasks, respectively, that are part of the original setup. for the full-scale comparison, we follow the original setup and keep the pretrained token representation models fixed, learning only layer weights and additional task-specific parameters on top of the weighted pretrained representations. we also conduct a smallscale study to compare the effect of fine-tuning on different span representations in terms of their relative ordering and for comparison with their non fine-tuned counterparts. overall, we find that the behavior of span representations tends to pattern according to whether they are based on information at the span boundaries versus using the entire span content. when pretrained models are frozen, we find that the choice of span representation is more important than the choice of pretrained model. when fine-tuning, the choice of span representation still has an impact on performance though it is much less pronounced than in the frozen case. although the best-performing method can vary greatly among tasks, we find in general that a span representation that simply takes the max over time is a reliable choice across tasks. ## span probing tasks we borrow four probing tasks applied by @xcite , namely constituent labeling, named entity labeling, semantic role labeling, and coreference arc prediction. we also introduce two new tasks: constituent detection and mention detection. the specific tasks are described below. constituent labeling is the task of predicting the non-terminal label (e.g., noun phrase, verb phrase, etc.) for a span corresponding to a constituent. constituent detection is the task of determining whether a span of words corresponds to a constituent (i.e., a nonterminal node) in the constituency parse tree of the input sentence. we introduce this task as a complement to the task of constituent labeling, to further evaluate the syntac-tic ability of the span representation methods. named entity labeling (nel) is the task of predicting the entity type of a given span corresponding to an entity, e.g., whether the span "german" in its sentence context refers to people, an organization, or a language. semantic role labeling (srl) is concerned with predicting the semantic roles of phrases in a sentence. in this probing task the locations of the predicate and its argument are given, and the goal is to classify the argument into its specific semantic roles (arg0, arg1, etc.). mention detection is the task of predicting whether a span represents a mention of an entity or not. for example, in the sentence "mary goes to the market", the spans "mary" and "the market" refer to mentions while all other spans are not mentions. the task is similar to named entity recognition @xcite , but the mentions are not limited to named entities. we introduce this task as it is the first step for coreference resolution @xcite , if the candidate mentions are not explicitly given. coreference arc prediction is the task of predicting whether a pair of spans refer to the same entity. for example, in the sentence "john is his own enemy", "john" and "his" refer to the same entity. ## models in this section, we first briefly describe the probing model, which is borrowed from @xcite with the extension to different span representations (figure 1 ), followed by details of the various span representation methods we compare in this work. ## analysis in this section we analyze the impact of span representation method with fixed pretrained encoders. ## related work many of the span representations that we consider here were proposed previously for specific tasks, such as the attention-weighted pooling of @xcite for coreference resolution; the endpointbased representation of @xcite and the "coherent" endpoint-based representation of @xcite for question answering; and combinations of differences and sums of endpoint representations for parsing and semantic role labeling @xcite . these are described in more detail in section 3.2. other recent work has considered pooling approaches such as the difference between endpoint representations @xcite or a concatenation of endpoint and attention-based representations @xcite . other approaches concatenate additional specialized feature vectors, such as the span length or position information @xcite @xcite . some work has also considered explicitly composing span representations via syntactic parse trees, such as recursive neural networks @xcite , and some unsupervised parsing models produce span representations as a byproduct of training @xcite . at the same time, there has been significant effort devoted to the related problem of learning representations for sentences or even longer texts @xcite @xcite @xcite . much of this work focuses on pooling over word representations, often finding that simple pooling operations like averaging perform surprisingly well @xcite . @xcite did a similar empirical study to ours in spirit, comparing a variety of pooling models for sentence representations across tasks. in this work we are mainly focusing on the models for computing span representations given pretrained token embeddings, but we also include a variety of pretrained contextual embeddings. one in particular, spanbert @xcite , was designed to enable improved span representations. while recent work has compared across pretrained contextual embeddings for representing spans @xcite , to our knowledge there has been no systematic comparison of methods for combining these contextual embeddings into span representations across a variety of tasks. ## conclusion we systematically compared multiple span representation methods, combined with various base embedding models, on various tasks. our analysis includes two new tasks that we propose to tease apart different aspects of span representations. when using fixed, pretrained encoders, we find that, although max pooling is a fairly reliable representation across tasks, the optimal span representation varies with respect to the syntactic and semantic nature of the task. finally, fine-tuning reduces the impact of span representation choice on performance, though it involves significant computational expense. our results are likely to be most useful for those without the computational capabilities to perform fine-tuning of large pretrained encoders, in which case there are significant differences among methods. in the table above we analyze the effect of learning the layerwise mixing weights vs simple averaging over layers in context of the coreference arc prediction task. attn-based models suffer the biggest drop with a drop of 0.6% absolute on average. among pretrained contextual embedding models, spanbert-large is hurt the most with a drop of 1% absolute on average. surprisingly, xl-net drops by only 0.1% on average even though its attention plots looked quite peaky for some of the span representations.