Aspect-based Document Similarity for Research Papers (2020)

Abstract: Traditional document similarity measures provide a coarse-grained distinction between similar and dissimilar documents. Typically, they do not consider in what aspects two documents are similar. This limits the granularity of applications like recommender systems that rely on document similarity. In this paper, we extend similarity with aspect information by performing a pairwise document classification task. We evaluate our aspect-based document similarity approach for research papers. Paper citations indicate the aspect-based similarity, i.e., the title of a section in which a citation occurs acts as a label for the pair of citing and cited paper. We apply a series of Transformer models such as RoBERTa, ELECTRA, XLNet, and BERT variations and compare them to an LSTM baseline. We perform our experiments on two newly constructed datasets of 172,073 research paper pairs from the ACL Anthology and CORD-19 corpus. According to our results, SciBERT is the best performing system with F1-scores of up to 0.83. A qualitative analysis validates our quantitative results and indicates that aspect-based document similarity indeed leads to more fine-grained recommendations.

## introduction recommender systems assist researchers in finding relevant papers for their work. when user feedback is sparse or unavailable, content-based approaches and corresponding document similarity measures are employed @xcite . recommender systems present a candidate document depending on whether it is similar or dissimilar to the seed document. this coarse-grained similarity assessment (similar or not) neglects the many facets that can make two documents similar. concerning the general concept of similarity, @xcite , and @xcite even argue that similarity is an ill-defined notion unless one can say to what aspects the similarity relates. in recommender systems for scientific papers, the similarity is often concerned with multiple facets of the presented research, e. g., method, findings @xcite . given that document similarity can differentiate research aspects, one could obtain tailored recommendations. for instance, a paper with similar methods but different findings could be recommended. such a recommender system would facilitate the discovery of analogies in research literature @xcite . we describe the underlying multiple aspect similarity in research papers as aspect-based document similarity. figure 1 contrasts aspect-based with aspect-free similarity @xcite . following the research paper example, aspect a 1 concerns findings and aspect a 2 methods (red and green in figure 1b ). in prior work @xcite , we propose to infer an aspect for document similarity formulating the problem as a multi-class classification of document pairs. we extend our prior work to a multi-label scenario and focus on scientific literature instead of wikipedia articles. similar to the work of @xcite and @xcite , we use citations as training signals. instead of using citations for binary classification (i. e., similar and dissimilar), we include the title of the section in which a citation occurs, as a label for a document pair. the section titles of citations describe the aspect-based figure 1 : recommender systems rely on similarity measures between a seed and the k most similar target documents (a). this neglects the aspects which make two or more documents similar. in aspect-based document similarity (b), documents are related according to the inner aspects connecting them (a 1 or a 2 ). similarity of citing and cited papers. our datasets originate from the acl anthology @xcite and cord-19 @xcite . in summary, our contributions are: (1) we extend traditional document similarity to aspect-based in a multi-label multi-class document classification task. (2) we demonstrate the aspect-based document similarity is well-suited for research papers. (3) we evaluate six transformer-based models and a baseline for the pairwise document classification task. (4) we publish our source code, trained models, and two datasets from the computational linguistics and biomedical domain, to foster further research. ## related work in the following, we discuss work on text similarity, recommendation, and applications of transformers. @xcite discuss the notion of similarity as often ill-defined in the literature and used as an "umbrella term covering quite different phenomena". @xcite also formalize what text similarity is and suggest content, structure, and style are the major dimensions inherent to text. for literature recommendation, the content and user information are the most predominant dimensions to consider @xcite . @xcite explore aspect-based document similarity as a segmentation task instead of a classification task. they segment the abstracts of collaborative and social computing papers into four classes, depending on their research aspects: background, purpose, mechanism, and findings. cosine similarity computed on segment representations allows the retrieval of similar papers for a specific aspect. @xcite apply the same segmentation approach on the cord-19 corpus @xcite . @xcite follow a related approach for citation recommendations. they classify sections into discourse facets and build document vectors for each facet. nevertheless, segmentation is a suboptimal alternative as it breaks the coherence of documents. with pairwise document classification, the similarity is aspect-based without sacrificing the document coherence. our experiments investigate transformer language models @xcite . bert @xcite , roberta @xcite , xlnet @xcite , and electra @xcite improve many nlp tasks, e. g., natural language inference @xcite and semantic textual similarity @xcite . @xcite demonstrate how bert models can be combined in a siamese network @xcite to produce embeddings that can be compared using cosine similarity. @xcite and @xcite explore bert for the classification of single documents with respect to sentiment or topic. @xcite and @xcite study domain-specific transformers for nlp tasks on scientific documents. moreover, cohan et al. (2020) are the first to use transformers to encode titles and abstracts of papers to generate recommendations. mohamed hassan et al. (2019) also use bert for recommendations, but only to encode paper titles. other recent recommender systems rely on other techniques such as introduction cites in sections seed target labels discussion … ## experiments we present our methodology (figure 2 ) for classifying the aspect-based similarity of research papers. ## results our results are divided into three parts: overall, label classes, and qualitative evaluation foot_3 . ## discussion in the experiments, scibert outperforms all other methods in the pairwise document classification. we observe in-domain pretraining and nsp objective often lead to higher f1-scores. transferring generic language models to a specific domain usually decreases the performance in our experiments. a possible explanation for this is the narrowly defined vocabulary in acl anthology or cord-19. @xcite and @xcite have also explored the transfer learning between domains with similar findings. covid-bert seems to be an exception as it yields lower results (micro-f1) than bert on cord-19 even though covid-bert was fine-tuned on cord-19. we observe the language model finetuning in covid-bert does not guarantee a higher performance compared to pretraining from scratch in scibert. however, covid-bert's authors provide too little information to give a proper explanation for its performance. apart from in-domain pretraining, the nsp objective has a positive effect on the models. all bert-based systems, which use nsp, outperform the models that excluded nsp (xlnet, roberta, and electra). we attribute the positive effect of nsp to its similarity to our task since both ## conclusion we applied pairwise multi-label multi-class document classification on scientific papers to compute aspect-based document similarity scores. we used section titles as aspects of paper and labeled citations occurring in these sections accordingly. the investigated models were trained to predict citations and the respected label based on the paper's title and abstract. we evaluated the transformer models bert, covid-bert, scibert, electra, roberta, and xlnet and a lstm baseline over two scientific corpora, i. e., acl anthology and cord-19. overall, scibert performed best in our experiments. despite the challenging task, scibert predicted the aspect-based document similarity with f1-scores of up to 0.83. scibert's performance motivates further research in this direction. it seems reasonable to include the aspect-based document similarity task as a new pretraining objective in the transformers architecture. this new objective could be integrated in similar fashion as the binary citation prediction objective @xcite proposed. as future work, we plan to integrate the aspect-based document similarity into a recommender system. thus, enabling a large user study to confirm our first indications that aspect-based document similarity indeed helps users to find more relevant recommendations. however, our extensive empirical analysis already demonstrates that transformers are well-suited to correctly compute the aspect-based document similarity for research papers. our datasets, code, and trained models are publicly available.