Dissecting Span Identification Tasks with Performance Prediction (2020)

Abstract: Span identification (in short, span ID) tasks such as chunking, NER, or code-switching detection, ask models to identify and classify relevant spans in a text. Despite being a staple of NLP, and sharing a common structure, there is little insight on how these tasks’ properties influence their difficulty, and thus little guidance on what model families work well on span ID tasks, and why. We analyze span ID tasks via performance prediction, estimating how well neural architectures do on different tasks. Our contributions are: (a) we identify key properties of span ID tasks that can inform performance prediction; (b) we carry out a large-scale experiment on English data, building a model to predict performance for unseen span ID tasks that can support architecture choices; (c), we investigate the parameters of the meta model, yielding new insights on how model and task properties interact to affect span ID performance. We find, e.g., that span frequency is especially important for LSTMs, and that CRFs help when spans are infrequent and boundaries non-distinctive.

## introduction span identification is a family of analysis tasks that make up a substantial portion of applied nlp. span identification (or short, span id) tasks have in common that they identify and classify contiguous spans of tokens within a running text. examples are named entity recognition @xcite , chunking @xcite , entity extraction @xcite , quotation detection @xcite , keyphrase detection @xcite , or code switching @xcite . in terms of complexity, span id tasks form a middle ground between simpler analysis tasks that predict labels for single linguistic units (such as lemmatization @xcite or sentiment polarity classification @xcite ) and more complex analysis tasks such as relation extraction, which combines span id with relation identification @xcite . due to the rapid development of deep learning, an abundance of model architectures is available for the implementation of span id tasks. these include isolated token classification models @xcite , probabilistic models such as hidden markov models @xcite , maximum entropy markov models @xcite , and conditional random fields @xcite , recurrent neural networks such as lstms @xcite , and transformers such as bert @xcite . though we have some understanding what each of these models can and cannot learn, there is, to our knowledge, little work on systematically understanding how different span id tasks compare: are there model architectures that work well generally? can we identify properties of span id tasks that can help us select suitable model architectures on a taskby-task basis? answers to these questions could narrow the scope of architecture search for these tasks, and could help with comparisons between existing methods and more recent developments. in this work, we address these questions by applying meta-learning to span identification @xcite . metalearning means "systematically observing how different machine learning approaches perform @xcite to learn new tasks much faster" @xcite , with examples such as architecture search @xcite and hyperparameter optimization @xcite . our specific approach is to apply performance prediction for span id tasks, using both task properties and model architectures as features, in order to obtain a better understanding of the differences among span id tasks. concretely, we collect a set of english span id tasks, quantify key properties of the tasks (such as how distinct the spans are from their context, and how clearly their boundaries are marked) and formulate hypotheses linking properties to performance (section 2). next, we describe relevant neural model architectures for span id (section 3). we then train a linear regressor as a meta-model to predict span id performance based on model features and task metrics in an unseen-task setting (section 4). we find the best of these architectures perform at or close to the state of the art, and their success can be relatively well predicted by the meta-model (section 5). finally, we carry out a detailed analysis of the regression model's parameters (section 6), gaining insight into the relationship between span id tasks and different neural model architectures. for example, we establish that spans that are not very distinct from their context are consistently difficult to identify, but that crfs are specifically helpful for this class of span id tasks. ## datasets, span types, and hypotheses we work with five widely used english span id datasets. all of them have non-overlapping spans from a closed set of span types. in the following, we discuss (properties of) span types, assuming that each span type maps onto one span id task. ## model architectures for span identification, we use the bio framework @xcite , framing span identification as a sequence labeling task. as each span type has its own b and i labels, and there is one o label, a dataset with n span types leads to a 2n + 1label classification problem for each token. we investigate a set of sequence labeling models, ranging from baselines to state-of-the-art architectures. we group our models by common components, and build complex models through combination of simpler models. except for the models using bert, all architectures assume one 300-dimensional glove embedding @xcite per token as input. baseline. as a baseline model, we use a simple token-level classifier. this architecture labels each token using a softmax classifier without access to sequence information (neither at the label level nor at the feature level). crf. this model uses a linear-chain conditional random field (crf) to predict token label se-quences @xcite . it can access neighboring labels in the sequence of predictions. lstm and lstm+crf. these architectures incorporate bi-directional lstms (bilstms) @xcite as components. the simplest architecture, lstm, passes the inputs through a 2-layer bilstm network, and then predicts token labels using a softmax layer. the lstm+crf architecture combines the bilstm network with a crf layer, training all weights simultaneously. these models can learn to combine sequential input and labeling information. bert and bert+crf. these architectures include the pre-trained bert language model (devlin et al., 2019) as a component. the simplest architecture in this category, bert, comprises a pre-trained bert encoder and a softmax output layer, which is trained while the bert encoder is fine-tuned. bert+crf combines a bert encoder with a linear-chain crf output layer, which directly uses bert's output embeddings as inputs. in this architecture, the crf layer is first trained to convergence while bert's weights are held constant, and then both models are jointly fine-tuned to convergence. as bert uses wordpiece tokenization @xcite , the input must be retokenized for bert architectures. bert+lstm+crf. this architecture combines all components previously mentioned. it first uses a pre-trained bert encoder to generate a sequence of contextualized embeddings. these embeddings are projected to 300 dimensions using a linear layer, yielding a sequence of vectors, which are then used as input for a lstm+crf network. as with bert+crf, we first train the non-bert parameters to convergence while holding bert's parameters fixed, and subsequently fine-tune all parameters jointly. handcrafted features. some studies have shown marked increases in performance by adding hand-crafted features (e.g. @xcite . we develop such features for our tasks and treat these to be an additional architecture component. for architectures with this component, a bag of features is extracted for each token (the exact features used for each dataset are enumerated in ## meta-learning model recall that our meta-learning model is a model for predicting the performance of the model architectures from section 3 when applied to span identification tasks from section 2. we model this task of performance prediction as linear regression, a well established framework for the statistical analysis of language data @xcite . the predictors are task properties, model architecture properties, and their interactions, and the dependent variable is (scaled) f 1 score. while a linear model is not powerful enough to capture the full range of interactions, its weights are immediately interpretable, it can be trained on limited amounts of data, and it does not overfit easily (see section 5.1). all three properties make it a reasonable choice for meta-learning. ## analysis we now investigate the linear regression coefficients of our performance prediction model to assess our hypotheses from section 2. to obtain a single model to analyze, we retrain our regression model on all data points, with no cross-validation. when analyzing these coefficients, one must consider main effects and interactions together. e.g., the main effect coefficient for lstms is negative, which seems to imply that adding an lstm will hurt performance. however, the lstm × [freq] and lstm × [boundary distinctness] interactions are both strongly positive, so lstms should help on frequent span types with high boundary distinctiveness. our main observations are the following: frequency helps, length hurts. the main effects of our span type predictors show mostly an expected pattern. frequency has a strong positive effect (frequent span types are easier to learn), while length has an even stronger negative effect (long span types are difficult). more distinct boundaries help performance as well. more surprising is the negative sign of the span distinctiveness predictor, which would mean that more distinct spans are more difficult to recognize. however, this might be due to the negative correlation between span distinctiveness and frequency (@xmath1.46 in standardized predictors) -less frequent spans are, by virtue of their rarity, more distinctive. bert is good for performance, especially with few examples. the presence of a bert component is the highest-impact positive predictor for model performance, with a positive coefficient of 1. this finding is not entirely surprising, given the recent popularity of bert-based models for span identification problems @xcite . furthermore, the strong negative value of the (bert × [freq]) predictor shows that bert's benefits are strongest when there are few training examples, validating our hypothesis about transfer learning. bert is also robust: largely independent of span or boundary distinctiveness effects. lstms require a lot of data. while the main effect of lstms is negative, this effect is again modulated by the high positive coefficient of the (lstm × [freq]) interaction. this means that their model predictors handcrafted (-0.11) crf 0.50 lstm -0.35 bert 1.00 span type predictors freq 0.40 length -0.49 span distinct. -0.22 boundary distinct. 0.16 model-span type interactions handcrafted × freq (0.05) length (-0.04) span distinct. (-0.09) boundary distinct. (0.09) crf × freq -0.33 length 0.19 span distinct. 0.34 boundary distinct. -0.30 lstm × freq 0.47 length 0.08 span distinct. (-0.09) boundary distinct. 0.22 bert × freq -0.43 length 0.13 span distinct. (0.04) boundary distinct. (-0.05) model-model interactions handcrafted × crf 0.10 lstm 0.05 bert -0.05 crf × lstm (-0.05) bert -0.24 lstm × bert -0.17 crfs help. after bert, the presence of a crf shows the second-most positive main effect on model performance. given the strong correlation between adjacent tags in a bio label sequence, it makes sense that a model capable of enforcing correlations in its output sequence would perform well. crfs can also exploit span distinctiveness well, presumably by the same mechanism. surprisingly, crfs show reduced effectiveness for highly frequent spans with distinct boundaries. we believe that this pattern is best considered as a relative statement: for frequent, well-separated span types crfs gain less than other model types. handcrafted features do not matter much. we find neither a significant main effect of handcrafted features, nor any significant interactions with span type predictors. interactions with model predictors are significant, but rather small. while a detailed analysis of architecture-wise f 1 -scores does show that some architectures, such as pure crfs, do seem to benefit more from hand-crafted features (see combining model components shows diminishing returns. all interactions between lstm, crf, and bert are negative. this demonstrates an overlap in these components' utility. thus, a simple "maximal" combination does not always perform best, as ## related work meta-learning and performance prediction are umbrella terms which comprise a variety of approaches and formalisms in the literature. we focus on the literature most relevant to our work and discuss the relationship. ## conclusion in this work, we considered the class of span identification tasks. this class contains a number of widely used nlp tasks, but no comprehensive analysis beyond the level of individual tasks is available. we took a meta-learning perspective, predicting the performance of various architectures on various span id tasks in an unseen-task setup. using a number of 'key metrics' that we developed to characterize the span id tasks, a simple linear regression model was able to do so at a reasonable accuracy. notably, even though bert-based architectures expectedly perform very well, we find that different variants are optimal for different tasks. we explain such patterns by interpreting the parameters of the regression model, which yields insights into how the properties of span id tasks interact with properties of neural model architectures. such patterns can be used for manual fine-grained model selection, but our meta-learning model could also be incorporated directly into automl systems. our current study could be extended in various directions. first, the approach could apply the same meta-learning approach to other classes of tasks beyond span id. second, a larger range of span type metrics could presumably improve model fit, albeit at the cost of interpretability. third, we only predict within-corpus performance, and corpus-level similarity metrics could be added to make predictions about performance in transfer learning.