Integrating knowledge graph embeddings to improve mention representation for bridging anaphora resolution (2020)

Abstract: Lexical semantics and world knowledge are crucial for interpreting bridging anaphora. Yet, existing computational methods for acquiring and injecting this type of information into bridging resolution systems suffer important limitations. Based on explicit querying of external knowledge bases, earlier approaches are computationally expensive (hence, hardly scalable) and they map the data to be processed into high-dimensional spaces (careful handling of the curse of dimensionality and overfitting has to be in order). In this work, we take a different and principled approach which naturally addresses these issues. Specifically, we convert the external knowledge source (in this case, WordNet) into a graph, and learn embeddings of the graph nodes of low dimension to capture the crucial features of the graph topology and, at the same time, rich semantic information. Once properly identified from the mention text spans, these low dimensional graph node embeddings are combined with distributional text-based embeddings to provide enhanced mention representations. We illustrate the effectiveness of our approach by evaluating it on commonly used datasets, namely ISNotes and BASHI. Our enhanced mention representations yield significant accuracy improvements on both datasets when compared to different standalone text-based mention representations.

## introduction an anaphor is an expression whose interpretation depends upon a previous expression in the discourse, an antecedent. a bridging anaphor is a special type of anaphor where there is non-identical or associative relation with its antecedent @xcite , as in the following example: in this case, the anaphor one employee depends on the antecedent starbucks for the complete interpretation and holds non-identical relationship with the antecedent, hence, a bridging anaphor. we here address the problem of learning from a set of anaphor-antecedent pairs a predictor capable of accurately identify such pairs in unseen texts. more precisely, if bridging resolution comprises two main tasks, bridging anaphora recognition and bridging anaphora resolution, we solely focus on the task of bridging anaphora resolution and assume that bridging anaphor recognition has already been performed. semantic information on anaphor-antecedent pairs plays a crucial role in resolving bridging anaphora. consider again the previous example: if the resolution system has the knowledge that starbucks is a company and companies have employees, then it is easy to establish the link between them. standard text-based features either hand-crafted or automatically extracted from word embeddings @xcite , are not sufficient for bridging resolution @xcite . earlier systems @xcite have proposed to extract this information from knowledge bases, the web, or raw text through queries of the form "x of y". the estimated number of occurrences in these sources gives the probability of relations between x and y. these types of queries were generalized by @xcite where all queries of the type "x preposition y", i.e. beyond this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ the mere "of" preposition, were considered. however, these approaches extract only shallow features, capturing relations between pair of nodes instead of taking advantage of broader information that is present in knowledge graphs. therefore, attempting to extend these strategies to take into account a larger amount of information on mentions may translate into learning problems where the input space is of high dimension, which might be a hurdle when dealing with moderate size datasets -for instance, the datasets that we consider here, i.e isnotes @xcite and arrau @xcite respectively contain 663 training pairs and 5512 training pairs. recently proposed approaches tried to remedy these shortcomings @xcite . hou learned embeddings on the pairs of nouns present in the text which are connected by prepositional or possessive structure (e.g. "x of y"). she creates "pseudo knowledge" by generating these noun-pairs and learn embeddings on these pairs. her approach is better at capturing fine-grained semantics than vanilla word embeddings such as word2vec @xcite , glove @xcite , etc. however, it still depends on the presence of the required noun-pairs in the corpus. the use of knowledge graphs, either manually or automatically constructed, can alleviate this problem as they contain general semantic and world-knowledge. we empirically demonstrate that embeddings constructed on these graphs indeed provide additional information and complement these text-based embeddings. in the present work, we propose to use low-dimensional graph node embeddings on knowledge graphs to capture semantic information. we use wordnet foot_0 @xcite as a knowledge graph in the experiments, though our approach can be extended to any knowledge graph. we hypothesize that the lowdimensional vectors learned on the nodes of wordnet graph capture lexical semantics such as hypernymy, hyponymy, meronymy, etc. as well as general relatedness between nodes. this way we eliminate the cumbersome task of manually designing features as well as the burden of querying. moreover, as we shall see, the low dimensionality of the embedding space does not go against its use with small datasets. but obtaining node embeddings for a mention is non-trivial a task, as it requires mapping a potentially ambiguous multi-token expression onto a specific node in the graph (synset in case of wordnet). this entails several key steps, such as: (i) mention normalization where the mention is mapped to a standardized form which might be present in the graph, (ii) handling the absent knowledge case where the referred entity is unavailable in the knowledge graph and possibly (iii) sense disambiguation in the presence of multiple senses for the mention. we propose simple yet effective heuristics to address these issues, as detailed in the coming sections. these knowledge graph embeddings are combined with distributional text-based embeddings to produce improved mention representations. we address the problem of bridging resolution as a ranking problem, where the trained model assigns a score to anaphor-candidate antecedent pairs, preferring this ranking approach over a classification perspective for it to be less sensitive to class-imbalance, and making it focused on learning relative scores. specifically, we train a ranking svm model to predict scores for anaphor-candidate antecedent pairs, an approach that has been successfully applied to the related task of coreference resolution @xcite . we observe that integrating node embeddings with text-based embeddings produces increased accuracy, substantiating the ability of graph node embeddings in capturing the semantic information. improving on the previous approach, @xcite proposed a generic query with all possible prepositions. their query is formulated as "x preposition y " instead of limiting to of preposition. they propose a global model as opposed to previous approaches that relied only on local features. in this model, they infer links globally instead of choosing from candidate set of the specific anaphor as they argue that the probability of noun phrase (np) being antecedent increases if it is already antecedent to another anaphor. their assumption is opposite to the local salience hypothesis of @xcite as the local models indirectly assume that the most salient candidate among the nearest context is the best suitable for antecedent. rule-based full bridging resolution system is proposed in @xcite where they devised rules for linking anaphors to antecedents. some of the rules as well as the corresponding features are acquired by querying the knowledge sources, albeit different queries such as a query to get a list of nouns which denote a part of buildingwall, window, or list of personal relationshusband, sister, etc. they also propose a learning-based system by converting the rules into features but observe slight gain. the work @xcite ) created word embeddings for bridging (embeddings pp) by exploring the syntactic structure of noun phrases (nps) to derive contexts for nouns in the glove model. she generalizes previous approaches of querying as her pp context model uses all prepositions for all nouns in big corpora. the deterministic approach proposed in @xcite is the extension to the work done in @xcite which creates new embeddings (embedding bridging) by combining embeddings pp and glove. her approach is efficient and solves the scalability and curse of dimensionality issues. but her approach depends on the presence of the np having a specific syntactic structure so that the algorithm can identify it as "x preposition y ". this algorithm misses those anaphor-antecedent pairs which do not possess this structure. the work @xcite uses neural networks trained on the relation classification tasks to get the semantic information between anaphor and antecedent. this information is integrated into the state-of-the-art systems for coreference and bridging resolution. the system fails at capturing broader semantic relations as only six semantic relations are predicted with neural networks, due to this they observe marginal improvement in the bridging resolution. all the previous works assume that the mentions are detected, i.e., noun phrases are presented and the task is to choose the correct np as an antecedent. this is discarded in the latest system, barqa @xcite . she casts bridging anaphora resolution as a question answering problem where answer produces antecedent for an anaphor. she also pointed out that most of the previous approaches relied only on the features of the antecedent-anaphor ignoring the context around them. however, she ignores any semantic information and relies on bert @xcite architecture to capture both contextual information as well as required common sense knowledge. knowledge graph embeddings graph embeddings represent graph (whole or sub-graph) or nodes with the lower dimensional vector. the work @xcite details a generic framework of the commonly used graph embedding algorithms. in recent times, embedding algorithms specifically for knowledge graphs have been proposed -rescal @xcite , distmult @xcite , complex @xcite , hole @xcite learn embeddings for knowledge graph completion, @xcite propose a2n neighborhood attention-based technique, @xcite embed relations with dihedral groups whereas @xcite employ graph attention network to acquire embeddings. in this work, we used wordnet as a knowledge graph so we are interested in the graph node embeddings learned particularly on wordnet @xcite @xcite . though, @xcite do not produce embeddings for senses present in wordnet as they encode corresponding words. however, path2vec @xcite produces embeddings for each sense present in wordnet by optimizing graph-based similarity metric. the use of knowledge graph embeddings to infuse common sense knowledge into nlp systems is becoming popular, and our work falls into this category. language model @xcite , domain-specific natural language inference (nli) @xcite , entity disambiguation @xcite have been some of the tasks where graph embeddings have been used. to the best of our knowledge, this is the first work where graph embeddings are used for bridging anaphora resolution. ## knowledge-aware mention representation in this paper, we propose a new, knowledge-aware mention representations for bridging resolution. these representations combine two components: (i) distributional embeddings learned from raw text data, and (ii) graph node embeddings learned from relational data obtained from a knowledge graph. specifically, the final representation v m for a mention m is obtained by concatenating the text-based contextual embeddings g m and the knowledge graph node embeddings h m : v @xmath0, h m ]. for the distributional embeddings g m , we use off-the-shelf word embeddings such as word2vec @xcite , glove @xcite , bert @xcite , or embeddings pp @xcite . except for bert, we average over embeddings of the mention's head word and common nouns appearing in the mention before the head, as mentioned in @xcite . with bert, mention embeddings are obtained by averaging over embeddings of all the words of the mention. however, obtaining knowledge graph-based embeddings h m for the mention is a much more challenging task, comprising different steps. before detailing those steps, we first briefly describe the knowledge graph -wordnet and how we compute node embeddings in the following paragraphs. knowledge graph is a graph with nodes being entities or abstract concepts and edges denoting the relation between them. a node in the knowledge graph can be a real-world entity such as a person, a place, etc. or can be an abstract concept such as a word, a sense, etc. a knowledge graph can be domain-specific (wordnet @xcite captures the semantic relation between words and meanings) or open domain (dbpedia @xcite for general-purpose knowledge). the central purpose of knowledge graphs is to store common sense knowledge in a structured format so that machines can easily access it. in this work, we have used wordnet as a knowledge repository but our approach is generic and can be applied with any other knowledge graph. wordnet @xcite primarily consists of synsets, i.e., a set of synonyms of words. the synsets which refer to the same concept are grouped together giving it a thesaurus-like structure. each synset consists of its definition and small example showing its use in a sentence. the synsets are connected with different relations such as synonymy, antonymy, hypernymy, hyponymy, meronymy, etc. in addition to the semantic knowledge, it also includes a bit of common sense knowledge such as real world entities like cities, countries and famous people. however, wordnet stores this knowledge as a database in its basic form, so a graph is constructed based on wordnet for further use. subsequently, the node embeddings learned on this graph will automatically capture the semantic information associated with the senses. we briefly discuss different wordnet node embedding algorithms used in our study. we use random walk and neural language model based embeddings @xcite , matrix factorization based wordnet embeddings @xcite and graph-similarity based path2vec @xcite embeddings. the important distinction between these methods is that the first two algorithms @xcite produce word embeddings and path2vec produces embeddings corresponding to each sense present in wordnet. the path2vec algorithm naturally encodes wordnet nodes as it actually produces embeddings for senses as opposed to @xcite algorithms as they conflate all the senses to produce word embeddings instead of generating embeddings for each sense while losing some finer semantic information in the process. the approach proposed by @xcite is based on the well-known neural language model continuous bag of words and skip-gram @xcite . the main idea is to produce artificial sentences from wordnet and to apply the language models on these sentences to produce word embeddings. for this, they perform random walk starting at any arbitrary vertex in wordnet, then map each wordnet sense to the corresponding word to produce an artificial sentence. each random walk produces a sentence, repeating this process several times gives a collection of sentences. finally, this collection of sentences is considered as the corpus for learning word embeddings. a different approach based on matrix factorization is taken in @xcite to produce embeddings. the procedure starts by creating the adjacency matrix m from wordnet graph. the element m ij in the matrix m is set to 1 if there exists any relation between words w i and w j . 2 furthermore, words which are not connected directly but via other nodes should also have an entry in the matrix, albeit with lower weights than 1. accordingly, matrix m g is constructed to get the overall affinity strength between words. in the analytical formulation, m g can be constructed from the adjacency matrix m as m @xmath1. following that, matrix m g is normalized to reduce the bias towards words which have more number of senses and finally a principal component analysis is applied to get vectors. the path2vec @xcite learns embeddings based on a pairwise similarity between nodes. the fundamental concept is that pairwise similarity between nodes of the graph should remain the same after their projection in the vector space. the model is flexible enough to consider any user-defined similarity measure while encoding. the objective function is designed to produce such embeddings for nodes which reduce the difference between actual graph-based pairwise similarity and vector similarity. it also preserves the similarity between adjacent nodes. formally, for the graph @xmath2, e) where v, e denote a set of vertices and edges, respectively, the objective is - ## ranking model let d be the given document containing @xmath3, m 2 , • • • , m nm }, n m number of mentions. let @xmath4, a 2 , • • • , a na } denote the set of all anaphors and a ⊂ m. let a be any anaphor in the set a and j be its position in the set m, then e a be the set of candidate antecedents for a which is defined as e @xmath5: m @xmath6, @xmath7. let t a and f a be the set of true antecedents and false candidate antecedents of a such that t a ∪ f @xmath8, t a ∩ f @xmath9. let each anaphor a is represented with the feature vector v a and candidate antecedent e represented with v e where @xmath10. then the goal is to predict score s(v a , v e ) between anaphor a and candidate antecedent e. the score denotes the possibility of anaphor a having bridging relation with the candidate antecedent e, so a higher score denotes a higher chance of e being true antecedent. the model is trained to reduce the ranking loss calculated based on the scores obtained between anaphor-candidate antecedents. the ranking strategy is fairly obviousfor an anaphor a high scoring candidate antecedent from e a is ranked higher than the low scoring one. let this prediction ranking strategy be r and true ranking is given by r * . for a candidate antecedent, if predicted rank is not the same as true rank then it is called discordant candidate, otherwise concordant. the difference between true and predicted ranking strategy can be measured with kendall's rank correlation coefficient -τ . formally, concordant c, discordant d candidates and τ are calculated as - inference we consider all the anaphors in the test document separately. for each anaphor, we consider all previously occurring mentions as candidate antecedents foot_4 and find out the compatibility score for each anaphor-candidate antecedent pair with the above ranking model. we apply best first strategy to choose the most appropriate antecedent from the list of candidate antecedents. in this strategy, the highest scoring pair is selected as anaphor-antecedent pair. formally, let a be any anaphor and e a denote a set of candidate antecedents for a. let s(a, e) be the score between a and e where @xmath11. let êa be the predicted antecedent of a which is given by -êa = argmax @xmath12, e) for the training data, we have positive samples where we know true anaphor-antecedent pairs but no negative samples. we generate these pairs by considering all the noun phrases (nps) which occur in our experiments section, we present results for different text-based embeddings -word2vec (wv), glove (gv), bert (be), embeddings pp (ep), bert + embeddings pp (bep) and the last columnshows the absence of text-based embeddings. also, in each row, wordnet node embeddings based on different algorithms, except the first row, are added -path2vec with lesk (pl), path2vec with averaged senses (pa), random walk based (rw) and wordnet embeddings (wnv). the other section of the table -sota, shows results with previously proposed systems -pairwise model iii (pmiii), mln model ii (mmii) @xcite , embeddings bridging (eb) @xcite , the combination of embeddings bridging and mln model (mmeb) and the latest system, barqa @xcite . the results with * are statistically signficant in comparison to the results based only on text embeddings with p-@xmath13. implementation we obtained pre-trained word2vec @xcite , glove @xcite , bert @xcite and embeddings pp @xcite embeddings. we used spanbert @xcite embeddings in our experiments as it gave better results in @xcite . also, we used pre-trained wordnet embeddings provided by respective authors of @xcite @xcite . in the case of path2vec @xcite , embeddings learned with different similarity measures such as -leacock-chodorow similarities @xcite ; jiang-conrath similarities @xcite ; wu-palmer similarities @xcite ; and shortest path similarities @xcite , are provided. we experimented with all the four similarity measures and found out that the shortest path based similarity measure produced better results most of the time, so we have used those embeddings in our experiments. we used python implementation of lesk algorithm from nltk foot_5 library to select the best sense from multiple senses of the mention. two sentences previous to mention and two sentences after the mention, including the sentence in which the mention occurs, are given to this algorithm as a context for a mention. both anaphor and candidate antecedent's embeddings are obtained as mentioned above, afterwards, element-wise product of these vectors is provided to the ranking svm. we also did preliminary experiments with the concatenation of the vectors but element-wise product gave better results. we used sv m rank @xcite implementation for our experiments. in the experiments with svm, we did grid search over @xmath14.001,0.01,0.1,1, 10,100 with the use of linear kernel. we also use random fourier features (rff) trick proposed by @xcite to approximate non-linear kernels. we found, use of non-linear kernels slightly improved results in comparison to linear kernels so reported only those results. we also varied different widow sizes of sentences -2,3,4 and all previous sentences, in addition to nps from the first sentence (salience), to get candidate antecedents for an anaphor. out of these settings, the window size of 2 and salience have yielded the best results which are reported here. ## results comparison between distributional and graph embeddings is shown in the following rows (2-4) of comparison between different wordnet embeddings we first examine the effectiveness of external knowledge without any text-based embeddings. these scores are noted in the last column of our experiments section against each wordnet graph node embeddings. the lower scores in this column in comparison with text-based embeddings reveal that the features learned with wordnet embeddings are not sufficient and should be complemented with the contextual features. this observation further substantiates our observation of higher scores with bert embeddings showing the importance of context ( ## conclusion we presented a simple approach of incorporating external semantic knowledge for bridging anaphora resolution. we combined contextual embeddings learned only on the text with the knowledge graph node embeddings. we establish the potency of knowledge graph embeddings with the experiments with the use of different wordnet graph embeddings on the isnotes and bashi datasets. though we apply a simplistic approach to solve mention normalization, absent knowledge resolution and sense disambiguation to obtain node embeddings, we achieve competitive results on both the datasets. moreover, this study opens up further investigation into the design of sophisticated methods to incorporate knowledge graph embeddings for bridging anaphora resolution such as improved mention normalization and sense disambiguation, incorporating knowledge from multiple knowledge sources.