Instance-Based Neural Dependency Parsing (2021)

Abstract: Interpretable rationales for model predictions are crucial in practical applications. We develop neural models that possess an interpretable inference process for dependency parsing. Our models adopt instance-based inference, where dependency edges are extracted and labeled by comparing them to edges in a training set. The training edges are explicitly used for the predictions; thus, it is easy to grasp the contribution of each edge to the predictions. Our experiments show that our instance-based models achieve competitive accuracy with standard neural models and have the reasonable plausibility of instance-based explanations.

## introduction while deep neural networks have improved prediction accuracy in various tasks, rationales underlying the predictions have been more challenging for humans to understand @xcite . in practical applications, interpretable rationales play a crucial role in driving humans' decisions and promoting human-machine cooperation @xcite . from this perspective, the utility of instance-based learning @xcite , a traditional machine learning method, has been realized again @xcite . instance-based learning is a method that learns similarities between training instances and infers a value or class for a test instance on the basis of similarities against the training instances. on the one hand, standard neural models encode all the knowledge in the parameters, making it challenging to determine what knowledge is stored and used for predictions @xcite . on the other hand, models with instance-based inference explicitly use training instances for predictions and can exhibit the instances that significantly contribute to the predictions. the instances play a role in an explanation to the question: why did the model make such a prediction? this type of explanation is called instance-based explanation @xcite @xcite , which facilitates the users' understandings of model predictions and allows users to make decisions with higher confidence @xcite . it is not trivial to combine neural networks with instance-based inference processes while keeping high prediction accuracy. recent studies in image recognition seek to develop such methods @xcite @xcite @xcite . this paradigm is called deep metric learning. compared to image recognition, there are much fewer studies on deep metric learning in natural language processing (nlp). as a few exceptions, @xcite and @xcite developed neural models that have an instance-based inference process for sequence labeling tasks. they reported that their models have high explainability without sacrificing the prediction accuracy. as a next step from targeting consecutive tokens, we study instance-based neural models for relations between discontinuous elements. to correctly recognize relations, systems need to capture associations between elements. as an example of relation recognition, we address dependency parsing, where systems seek to recognize binary relations between tokens (hereafter edges). traditionally, dependency parsers have been a useful tool for text analysis. an unstructured text of interest is parsed, and its structure leads users to a deeper understanding of the text. by successfully introducing instance-based models to dependency parsing, users can extract dependency edges along with similar edges as a rationale for the parse, which further helps the process of text analysis. in this paper, we develop new instance-based neural models for dependency parsing, equipped with two inference modes: (i) explainable mode and (ii) fast mode. in the explainable mode, our models make use of similarities between the candidate edge and each edge in a training set. by looking at the similarities, users can quickly check which training edges significantly contribute to the prediction. in the fast mode, our models run as fast as standard neural models, while general instance-based models are much slower than standard neural models because of the dependence on the number of training instances. the fast mode is motivated by the actual situation: in many cases, users want only predictions, and when the predictions seem suspicious, they want to check the rationales. thus, the fast mode does not offer rationales, but instead enables faster parsing that outputs exactly the same predictions as the explainable mode. users can freely switch between the explainable and fast modes according to their purposes. this property is realized by taking advantage of the linearity of score computation in our models and avoids comparing a candidate edge to each training edge one by one for computing the score at test time (see section 4.4 for details). our experiments on multilingual datasets show that our models can achieve competitive accuracy with standard neural models. in addition, we shed light on the plausibility of instance-based explanations, which has been underinvestigated in dependency parsing. we verify whether our models meet a minimal requirement related to the plausibility @xcite . additional analysis reveals the existence of hubs @xcite , a small number of specific training instances that often appear as nearest neighbors, and that hubs have a terrible effect on the plausibility. our main contributions are as follows: ## dependency parsing framework we adopt a two-stage approach @xcite : we first identify dependency edges (unlabeled dependency parsing) and then classify the identified edges (labeled dependency parsing). more specifically, we solve edge identification as head selection and solve edge classification as multi-class classification. foot_2 ## instance-based scoring methods for the scoring functions in eqs. 1 and 4, we describe our proposed instance-based models. ## conclusion we have developed instance-based neural dependency parsing systems, each of which consists of our edge identification model and our label classification model (section 4). we have analyzed them from the perspectives of the prediction accuracy and the explanation plausibility. the first analysis shows that our instance-based systems and achieve competitive accuracy with weightbased neural ones (section 6.1). the second indicates that our instance-based systems using the cosine similarity (l2-normalization for edge representations) meet the minimal requirement of plausible explanations (section 6.2). the additional analysis reveals that when using the dot product, hubs emerge, which degrades the plausibility (section 6.3). one interesting future direction is investigating the cause of hubs' occurrence in more detail. another direction is using the learned edge representations in downstream tasks, such as semantic textual similarity.