Dual Attention Model for Citation Recommendation (2020)

Abstract: Based on an exponentially increasing number of academic articles, discovering and citing comprehensive and appropriate resources has become a non-trivial task. Conventional citation recommender methods suffer from severe information loss. For example, they do not consider the section of the paper that the user is writing and for which they need to find a citation, the relatedness between the words in the local context (the text span that describes a citation), or the importance on each word from the local context. These shortcomings make such methods insufficient for recommending adequate citations to academic manuscripts. In this study, we propose a novel embedding-based neural network called “dual attention model for citation recommendation (DACR)” to recommend citations during manuscript preparation. Our method adapts embedding of three semantic information: words in the local context, structural contexts, and the section on which a user is working. A neural network model is designed to maximize the similarity between the embedding of the three input (local context words, section and structural contexts) and the target citation appearing in the context. The core of the neural network model is composed of self-attention and additive attention, where the former aims to capture the relatedness between the contextual words and structural context, and the latter aims to learn the importance of them. The experiments on real-world datasets demonstrate the effectiveness of the proposed approach.

## introduction when writing an academic paper, one of the most frequent questions considered is: "which paper should i cite at this place?" based on the massive number of papers being published, it is impossible for a researcher to read every article that might be relevant to their study. thus, recommending a handful of useful citations based on the contents of a working draft can significantly alleviate the burden of writing a paper. an example of the application scenario is demonstrated in figure 1 . currently, many scholars rely on "keyword searches" on search engines, such as google scholar 2 and dblp 3 . however, keyword-based systems often generate unsatisfying results, because query words may not convey adequate information to reflect the context that needs to be supported @xcite . researchers in various fields have proposed various methods to solve this problem. for example, studies in @xcite @xcite @xcite considered recommendations based on a collection of seed papers, and @xcite proposed methods using meta-data, such as authorship information, titles, abstracts, keyword lists, and publication years. however, when applying such methods to real-world paper-writing tasks, there is a lack of consideration for the local context of a citation within a draft, which can potentially lead to suboptimal results. context-based recommendations adopt a more practical concept that generates potential citations for an input context @xcite this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . based on the context-based methodology, the hyperdoc2vec @xcite proposed an embedding framework which further considers embedding with information of citation link between the local context in a citing paper and the content in a cited paper. our previous study @xcite adapted the structural contexts in addition to citation link to further improve the recommendation performances. context-based approaches could be potentially applicable in the real-world paper-writing process. however, the studies mentioned above still fail to take into consideration a number of essential characteristics of academic papers, which limits their usefulness. adequate recommendations of citations for a manuscript should capture the relatedness and importance of words and cited articles in the context which needs citations, as well as the purpose of the section on which the writer is currently working. to this end, we propose a novel embedding-based neural network called dual attention model for citation recommendation (dacr) that is designed to capture the relatedness and importance of words in the context which needs citations and structural contexts in the manuscript, as well as the section for which the user is working. the core of the proposed neural network is composed of two attention mechanisms, namely self-attention and additive attention. the former captures the relatedness between contextual words and structural contexts, and the latter learns the importance of contextual words and structural contexts. additionally, the proposed model embeds sections into an embedding space and utilizes the embedded sections as additional features for recommendation tasks. ## dual attention model for citation recommendation an overview of the proposed dacr approach is presented in figure 2 . dacr is composed of two main components: a context encoder (section 4.1) for encoding contextual words, sections, and structural contexts into a fixed-length vector and a citation encoder (section 4.2) for predicting the probability of a target citation. ## experiments we evaluated the recommendation performance of our model and five baseline models on two datasets, namely dblp and acl anthology @xcite . the recall, mean average precision (map), mean reciprocal rank (mrr), and normalized discounted cumulative gain (ndcg) are reported for a comparison of the models. these values are summarized in ## conclusions and future work in this study, we proposed a citation recommendation model with dual attention mechanisms. this model aims to simplify real-world paper-writing tasks by alleviating the issue of information loss in existing methods. our model considers three types of essential information: section for which a user is working and need to insert citations, relatedness between the local context words and structural contexts, and their importance. the core of the proposed model is composed of two attention mechanisms: selfattention for capturing relatedness and additive attention for learning importance. extensive experiments demonstrated the effectiveness of the proposed model in designed scenarios intended to mimic the real world scenarios as well as the efficiency of the proposed neural network. in future work, we will first attempt to improve the accuracy of recognizing section headers to improve the usability and performance of the algorithm. second, we will include additional paper-related information in the model, such as word positions. third, we will explore more sophisticated neural network architectures to improve accuracy and reduce the training time of the model.