Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition (2020)

Abstract: Interpretable rationales for model predictions play a critical role in practical applications. In this study, we develop models possessing interpretable inference process for structured prediction. Specifically, we present a method of instance-based learning that learns similarities between spans. At inference time, each span is assigned a class label based on its similar spans in the training set, where it is easy to understand how much each training instance contributes to the predictions. Through empirical analysis on named entity recognition, we demonstrate that our method enables to build models that have high interpretability without sacrificing performance.

## introduction neural networks have contributed to performance improvements in structured prediction. instead, the rationales underlying the model predictions are difficult for humans to understand @xcite . in practical applications, interpretable rationales play a critical role for driving human's decisions and promoting human-machine cooperation @xcite . with this motivation, we aim to build models that have high interpretability without sacrificing performance. as an approach to this challenge, we focus on instance-based learning. instance-based learning @xcite is a machine learning method that learns similarities between instances. at inference time, the class labels of the most similar training instances are assigned to the new instances. this transparent inference process provides an answer to the following question: which points in the training set most closely resemble a test point or influenced the prediction? this is categorized into example-based explanations @xcite . recently, despite its preferable property, it has received little attention and been underexplored. this study presents and investigates an instancebased learning method for span representations. a span is a unit that consists of one or more linguistically linked words. why do we focus on spans instead of tokens? one reason is relevant to performance. recent neural networks can induce good span feature representations and achieve high performance in structured prediction tasks, such as named entity recognition (ner) @xcite , constituency parsing @xcite , semantic role labeling (srl) @xcite and coreference resolution @xcite . another reason is relevant to interpretability. the tasks above require recognition of linguistic structure that consists of spans. thus, directly classifying each span based on its representation is more interpretable than token-wise classification such as bio tagging, which reconstructs each span label from the predicted token-wise bio tags. our method builds a feature space where spans with the same class label are close to each other. at inference time, each span is assigned a class label based on its neighbor spans in the feature space. we can easily understand why the model assigned the label to the span by looking at its neighbors. through quantitative and qualitative analysis on ner, we demonstrate that our instancebased method enables to build models that have high interpretability and performance. to sum up, our main contributions are as follows. neural models generally have a common technical challenge: the black-box property. the rationales underlying the model predictions are opaque for humans to understand. many recent studies have tried to look into classifier-based neural models @xcite @xcite . in this paper, instead of looking into the black-box, we build interpretable models based on instance-based learning. before the current neural era, instance-based learning, sometimes called memory-based learning @xcite , was widely used for various nlp tasks, such as part-of-speech tagging @xcite , dependency parsing @xcite and machine translation @xcite . for ner, some instance-based models have been proposed @xcite @xcite . recently, despite its high interpretability, this direction has not been explored. one exception is @xcite , which used instance-based learning of token representations. due to bio tagging, it faces one technical challenge: inconsistent label prediction. for example, an entity candidate "world health organization" can be assigned inconsistent labels such as "b-loc i-org i-org," whereas the groundtruth labels are "b-org i-org i-org." to remedy this issue, they presented a heuristic technique for encouraging contiguous token alignment. in contrast to such token-wise prediction, we adopt span-wise prediction, which can naturally avoid this issue because each span is assigned one label. ner is generally solved as (i) sequence labeling or (ii) span classification. 2 in the first approach, token features are induced by using neural networks and fed into a classifier, such as conditional random fields @xcite @xcite . one drawback of this approach is the difficulty dealing with nested entities. 3 by contrast, the span classification approach, adopted in this study, can straightforwardly solve nested ner @xcite @xcite . 4 3 instance-based span classification 3.1 ner as span classification ner can be solved as multi-class classification, where each of possible spans in a sentence is assigned a class label. as we mentioned in section 2, this approach can naturally avoid inconsistent label prediction and straightforwardly deal with nested entities. because of these advantages over tokenwise classification, span classification has been gaining a considerable attention @xcite . formally, given an input sentence of t words @xmath0, w 2 , . . . , w t ), we first enumerate possible spans s(x), and then assign a class label @xmath1. we will write each span as @xmath2, b), where a and b are word indices in the sentence: here, the possible spans in this sentence are s(x) = {(1, 1), (1, 2), (1, 3), . . . , (4, 5), (5, 5)}. "franz kafka," denoted as @xmath3, 2), is assigned the person type entity label (@xmath4. note that the other non-entity spans are assigned the null label (@xmath5. for example, "a novelist," denoted as @xmath6, 5), is assigned null. in this way, the null label is assigned to non-entity spans, which is the same as the o tag in the bio tag set. the probability that each span s is assigned a class label y is modeled by using softmax function: typically, as the scoring function, the inner product between each label weight vector w y and span feature vector h s is used: the score for the null label is set to a constant, score(s, @xmath7, similar to logistic regression @xcite . for training, the loss function we minimize is the negative log-likelihood: ## experiments 4.1 experimental setup data we evaluate the span models through two types of ner: (i) flat ner on the conll-2003 dataset @xcite and (ii) nested ner on the genia dataset foot_5 @xcite . we follow the standard trainingdevelopment-test splits. baseline we use a classifier-based span model (section 3.1) as a baseline. only the difference between the instance-based and classifier-based span models is whether to use softmax classifier or not. ## conclusion we presented and investigated an instance-based learning method that learns similarity between spans. through ner experiments, we demonstrated that the models build by our method have (i) competitive performance with a classifier-based span model and (ii) interpretable inference process where it is easy to understand how much each training instance contributes to the predictions.