Virtual Citation Proximity ( VCP ): Empowering Document Recommender Systems by Learning a Hypothetical In-Text Citation-Proximity Metric for Uncited Documents (2020)

Abstract: The relatedness of research articles, patents, court rulings, web pages, and other document types is often calculated with citation or hyperlink-based approaches like co-citation (proximity) analysis. The main limitation of citation-based approaches is that they cannot be used for documents that receive little or no citations. We propose Virtual Citation Proximity (VCP), a Siamese Neural Network architecture, which combines the advantages of co-citation proximity analysis (diverse notions of relatedness / high recommendation performance), with the advantage of content-based filtering (high coverage). VCP is trained on a corpus of documents with textual features, and with real citation proximity as ground truth. VCP then predicts for any two documents, based on their title and abstract, in what proximity the two documents would be co-cited, if they were indeed co-cited. The prediction can be used in the same way as real citation proximity to calculate document relatedness, even for uncited documents. In our evaluation with 2 million co-citations from Wikipedia articles, VCP achieves an MAE of 0.0055, i.e. an improvement of 20% over the baseline, though the learning curve suggests that more work is needed.

## introduction calculating document relatedness is key in creating recommender systems for digital libraries (we focus on research paper recommenders -our work is, however, equally applicable to patents, websites, court rulings and other documents with hyperlinks, citations respectively). recommender systems in digital libraries calculate relatedness of research articles typically via content-based filtering or hyperlink/citation-based approaches @xcite @xcite . citation-based approaches consider documents as related that reference the same documents (bibliographic coupling), that are co-cited by other documents or that are otherwise connected in the citation graph @xcite . citation-based approaches may recommend more diverse items than content-based filtering, as citations can be made for various reasons @xcite @xcite . for instance, two documents can be co-cited because they address the same research problem; use the same methodology (to solve different problems); or two documents may be cocited for less predictable reasons. today's textbased methods can hardly distinguish such diverse types of relatedness. instead, text-based methods generally consider two documents as related the more terms they have in common 1 . a particularly promising citation-based approach is citation proximity analysis (cpa) @xcite , which is illustrated in figure 1 . cpa considers documents as the more related, the closer the distance in which they are co-cited. for instance, in the example, the citing document cites document a and document b in the same sentence. document c is cited in a different paragraph. hence, a and b are more related than a and c (or b and c). cpa out-performs standard co-citation analysis by up to 95% @xcite and has successfully been used with research articles @xcite @xcite , wikipedia @xcite , web pages @xcite , mind-maps @xcite and authors @xcite . the downside of cpa is that it can be only be applied to documents that are (co-)cited. most research articles, however, are @xcite . a citing document cites the three documents a, b, and c. documents a and b are cited within the same sentence and are hence strongly related. documents a and c, as well as documents b and c, are each cited within different paragraphs. hence, they are considered as less strongly related to each other. a recommender system that receives document b as input, and that should recommend the most related document, would recommend document a. we propose foot_1 , implement and evaluate a novel approach that we name 'virtual citation proximity' (vcp). we hypothesize that vcp combines the advantages of co-citation proximity analysis (diverse notions of relatedness / high recommendation effectiveness), with the advantage of content-based filtering (high coverage). hence, we expect that vcp advances the state-of-the-art in related-document calculations for search engines and recommender systems significantly. ## virtual citation proximity (vcp) virtual citation proximity (vcp) predicts in which distance two documents -that are not co-citedwould be co-cited if they were co-cited. this pre-dicted proximity can then be used in the same way as real co-citation proximity to calculate document relatedness. at an abstract level, the idea behind vcp is that there is an inherent concept of relatedness between articles. this inherent relatedness can be described either through text or co-citations. as both, text and citations, eventually refer to the same relatedness, the text and citation are kind of a 'siamese twin'. we propose to implement vcp via artificial neural networks that are trained with textual features -e.g. terms or word embeddings from the title or abstract -as input, and real citation proximity as target. in other words, we feed a neural network with pairs of documents of which we know how strongly they are related (expressed by the real proximity of their co-citations). the network then learns a similarity function that predicts based on the text the degree to which the two documents are related -even if the two documents have no terms or word embeddings in common. we hypothesize that a neural network will be able to learn the diverse types of relatedness inherent to co-citations. once the network is trained, it receives the text of two documents as input, and predicts in what proximity these two documents would be co-cited if they were co-cited. vcp can be applied to all document pairs in a corpus that contain a title (and abstract), i.e. typically all document in a corpus (100% coverage). if the predictions of vcp are precise, a recommender system based on vcp would be as effective as a system based on real citation proximity, but with a coverage as high as content-based filtering (100%). although virtual citation proximity is based on textual features as input, we hypothesise that vcp will create recommendations similar to those based on real citation-proximity, since the machine learning algorithm is trained on real citation proximity as ground truth. with the recent advances in (deep) machine learning we hypothesise that a (deep) machine-learning algorithm will be able to detect hidden layers in the text. these will allow determining what makes two documents related, more reliable than the typical assumption in textbased approaches that two documents are related when they share the same terms or embeddings. ## related work virtual citation proximity trains a machine learning model with real citation proximity as ground truth / target, and to the best of our knowledge we are first to do this. the method that is closest to using citation-proximity as ground truth for machine learning is using expert judgements (or knowledge bases) as ground truth, e.g. mesh, acm ccs, or dmoz @xcite . for instance, the mesh classification is a classification tree that represents the major fields and subfields in the biomedical domain. mesh was created by medical experts and biomedical manuscripts are often classified with mesh, i.e. manuscripts are assigned to one of the mesh categories, whereas two documents in the same category are considered to be related, and can be used either for training machine learning models or evaluating recommendation approaches @xcite . machine learning algorithms can infer from the existing documents in a category, which textual features make a document likely to belong to a certain category. new documents can then automatically be classified based on their text @xcite , there are disadvantages to using expert classifications like mesh, when compared to citations and vcp respectively. first, expert classifications are often one-dimensional, i.e. they provide only one type of relatedness (typically, the overall topic a research article is about). second, most expert classification schemes allow documents to be in few categories only, and they focus on one field (e.g. medicine or computer science). especially with today's increasingly interdisciplinary work, this is often not enough to adequately find all related documents. third, classification schemes typically have a limited number of categories (a few thousand at most). this means, in large collections, categories contain thousands of documents that are somewhat related to each other but only at a relatively broad level. fourth, classifications are often static, i.e. articles are classified at the time of publication. if a classification scheme is changed, the papers are not updated or re-classified. finally, for many domains, expert classifications simply do not exist. with vcp, the problems could be overcome. (virtual) citation proximity (1) covers many types of relatedness; (2) allows documents to be in unlimited numbers of co-citation clusters; in recent years advances in deep-learning have shown the ability to identify complex patterns in text based data in areas such as translation @xcite and sentiment analysis @xcite . a document embedding @xcite is an embedding representing an entire document trained using a paragraph embedding model. document embedding vectors have been shown to be superior to other text representations such as bag-of-words as they take into account the relative positions of the words in the text, although experimental they may be an interesting feature representation to train vcp. overall, papers with success in using machine learning for dealing with larger passages of text more limited in number @xcite , compared to longer texts @xcite . some relevant research was found in the areas of news article recommender systems @xcite . ## results and discussion all four models achieved relatively low maes between 0.0059 (sequential 1d cnn + lstm) and 0.0055 (siamese lstm + lstm; siamese cnn + lstm, no dropout) (figure 5 ). all three siamese neural networks outperformed the simple sequential model cnn+lstm. the differences among the three siamese architectures are statistically not significant. all four models performed statistically significant better (p¡0.01; two-tailed t-test) than the baseline, i.e. the mean cpi in the dataset (@xmath0.0069). the low maes must be seen with some skepticism. the average of the actual cpi values in the dataset was 0.0069 with data skewed towards smaller values. hence, an mae of e.g. 0.0055 is promising (20% lower, i.e. better, than the mean cpi) but not as good as it may seem on first glance. the learning curves of the four vcp approaches indicates that citation proximity could not be learned very effectively. figure 6 shows the training and validation error rates of the siamese cnn + lstm model over 50 epochs. the validation error shows that no real learning occurs after the first epoch. overall, our result, i.e. a 20% improvement over the trivial 'mean' baseline, is promising but more research is needed to confirm the effectiveness of virtual citation proximity. in the current experiment, we used the average cpi of document pairs as target, but alternatives such as the minimum or maximum cpi might be easier to learn for a siamese network. also, there were many documents with low cpi values in the corpus, which might have introduced noise. in future work, we would focus on documents with higher cpi values as we expect their signal to be stronger. we also plan to use more than 200 words in future experiments, as more words might contain more semantic meaning of why a document was cited. maybe most importantly, virtual citation proximity needs to be evaluated in more recommender-system specific experiments. so far, we 'only' predicted citation distance. the key question, however, is how good vcp-based recommendations can be, i.e. how precise they need to be to contribute to business value @xcite . it will also be interesting to see how vcp compares with contentbased filtering, citation-based approaches, and machine learning models trained on expert opinions as ground truth. while our initial results are 'only' good, we see an enormous potential in virtual citation proximity for improving recommender systems for research papers, web pages, patents, and other document types. we are confident that vcp could become a new state-of-the-art approach for research paper recommender systems that brings citation-based recommendation effectiveness to the community, applicable to all textual documents. in the best case, vcp might even outperform citation based approaches as vcp learns from both terms and citations and hence vcp might be able to learn semantic concepts in a completely new way beyond traditional citation and content analysis.