An Empirical Study on Finding Spans (2022)

Abstract: We present an empirical study on methods for span finding, the selection of consecutive tokens in text for some downstream tasks. We focus on approaches that can be employed in training end-to-end information extraction systems, and find there is no definitive solution without considering task properties, and provide our observations to help with future design choices: 1) a tagging approach often yields higher precision while span enumeration and boundary prediction provide higher recall; 2) span type information can benefit a boundary prediction approach; 3) additional contextualization does not help span finding in most cases.

## introduction various information extraction (ie) tasks require a span finding component, which either directly yields the output or serves as an essential component of downstream linking. in named entity recognition (ner), spans in text are detected and typed; coreference resolution (re) requires mention spans; mention spans are linked when performing relation extraction (re), and in event extraction this also requires detection of trigger spans. in extractive question answering (qa), a span in a passage is detected to be presented as the answer to a given question. following the proliferation of large pre-trained models @xcite @xcite , recent approaches to span finding can be roughly divided into three different types: as tagging, span enumeration, or boundary prediction (see §2, a: the choice depends on the downstream task: tagging generally has higher precision, but span enumeration or boundary prediction has higher recall. for most cases, boundary prediction is preferable to span enumeration. tagging performs much better on a masked language model pretrained encoder (e.g., roberta) than an encoder-decoder pretrained model (e.g., t5). a: for downstream ie tasks, tagging and span enumeration approaches prefer untyped tags, but boundary prediction heavily relies on type information to obtain good performance. ## background we assume a pre-trained base model is in place (e.g. bert @xcite , t5 @xcite ). the encoding for each token 𝑥 𝑖 is denoted as x 𝑖 ∈ r 𝑑 . for further analyses, we focus on roberta @xcite and t5 since these represent two classes of pretrained models: one with an encoder trained with reconstruction loss; the other with both an encoder and a decoder. tagging span selection can be reduced to a sequence tagging problem, usually under the bio scheme. such tagging problems can be modeled using a linear-chain conditional random field (crf; @xcite , with tags either typed or untyped (for example, in ner tags may be labeled with entity types b-person, i-location, or without, b, i). in recent work, features input to crf may be hand-crafted features or predominantly outputs of a neural network. note that reducing span finding to a tagging problem does not allow the system to produce overlapping spans. there exist methods to address these (e.g., for the task of nested ner), but we leave the analysis of these methods for future work. ## experimental setup we perform all our experiments on roberta base and the encoder part of t5 base (t5 enc base ). each number reported is an average of 3 runs with different random seeds. each run is trained with a single quadro rtx 6000 gpu with 24gb memory. ## conclusions we identified and investigated three common span finding methods in the nlp community: tagging, span enumeration, and boundary prediction. through extensive experiments, we found that there is not a single recipe that is best for all scenarios. the design choices on downstream tasks rely on specific task properties, specifically the tradeoff between precision and recall. we suggest that precision-focused tasks consider tagging or boundary prediction, and recall-focused (such as coreference resolution) tasks consider span enumeration or boundary prediction. we further examined two commonly used tricks to improve span finding performance, i.e., adding span type information during the training and adding additional contextualization with an rnn on top. we observed that boundary prediction on ie tasks heavily relies on type information, and adding additional contextualization mostly does not help span finding. architectures will continue to evolve and models will continue to grow in size, which may lead to different conclusions on the relative benefits of approaches. still, the fundamental task of isolating informative spans of text will remain. we hope this study helps inform system designers today with existing models and still in the future as a starting point for further inquiry.