Is the ranking of P ub M ed similar articles good enough? An evaluation of text similarity methods for three datasets (2023)

Abstract: The use of seed articles in information retrieval provides many advantages, such as a longercontext and more details about the topic being searched for. Given a seed article (i.e., a PMID), PubMed provides a pre-compiled list of similar articles to support the user in finding equivalent papers in the biomedical literature. We aimed at performing a quantitative evaluation of the PubMed Similar Articles based on three existing biomedical text similarity datasets, namely, RELISH, TREC-COVID, and SMAFIRA-c. Further, we carried out a survey and an evaluation of various text similarity methods on these three datasets. Our experiments considered the original title and abstract from PubMed as well as automatically detected sections and manually annotated relevant sentences. We provide an overview about which methods better performfor each dataset and compare them to the ranking in PubMed similar articles. While resultsvaried considerably among the datasets, we were able to obtain a better performance thanPubMed for all of them. Datasets and source codes are available at: https://github.com/mariananeves/reranking

## introduction tools for searching for relevant publications in the biomedical literature need to rank results with respect to their relevance to the user @xcite . however, different users are often interested in different aspects of the publications. a study of the search logs from pubmed showed many interesting aspects of the user interaction with the tool @xcite . since 80% of the viewed abstracts derived from the top 20, a good ranking algorithm is important. further, queries are rather short (less than four tokens) and usually composed of a mix of semantic associations, including abbreviations, diseases, chemicals, author names, etc. however, in some situations, such as when searching for a particular research goal, it is a complex task to precisely define the search using just a couple of words. pubmed similar articles allow a search based on seed articles, i.e., it provides a pre-compiled list of articles that are similar to the given seed article foot_0 @xcite . it is a valuable resource with many applications, e.g., for building clusters of articles @xcite , entity networks @xcite , or similarity-based datasets @xcite . in comparison with keywords-based queries, seed articles provide a larger context, and potentially, more information about the subject being searched, e.g., details about the research goal, or long names for some abbreviations. seed articles have been previously used for a variety of tasks in information retrieval, such as in the construction of bag of words @xcite , recommendation systems @xcite , or in systematic reviews @xcite . the pubmed similar articles can easily be queried with the entrez programming utilities (eutilities) foot_1 . the methods behind the pubmed similar articles are based on a probabilistic topic-based model for content similarity (called "pmra") @xcite . the developers of the function evaluated their method on the data of the trec 2005 genomics track @xcite and compared it to bm25 @xcite . in this manuscript we aimed at evaluating the ranking from pubmed similar articles for a set of available text similarity datasets. our goal was to analyze the performance of this function in comparison with state-of-the-art algorithms for text similarity. we did neither train nor fine-tune any of the methods we used, i.e., we relied only on methods which either do not need to be trained (e.g., bm25) or which were pre-trained ones (e.g., language models). further, in order to be able to compare results to pubmed similar articles, we relied only on datasets based on pubmed. in our experiments, we considered the original abstracts in pubmed as well as automatically selected sections and manually annotated facets (relevant sentences). we are only aware of three previous similar evaluations of the pmra algorithm: (i) the relish database, in which the authors compared pmra to bm25 and tf-idf for a large collection of more than 180k articles and more than 3k seed articles @xcite ; (ii) the smafira-c dataset, in which an evaluation was carried out for three seed articles @xcite ; and (iii) an evaluation for seven seed articles with the focus on the abstracts' sections @xcite . a couple of previous projects also carried out an evaluation on some of the datasets that we used @xcite . however, they did not aim at evaluating the performance of the pmra algorithm. the contributions of our work are the following: (a) a short review of the various text similarity methods and datasets; (b) an evaluation of these methods on the three selected datasets, and based on a variety of semantic features; (c) a comparison of these methods to pubmed similar articles; and (d) making available the derived datasets, manual annotations, and source code of our experiments and evaluation. ## datasets we selected datasets composed of pubmed abstracts and which included annotations with respect to relevance or similarity to a particular seed article. therefore, we skipped datasets that did not comply with one or more of these conditions, such as cs-fcube @xcite , which is composed of publications from the areas of computational linguistics and machine learning. we considered each dataset in two settings: (a) as originally released by the authors, and (b) their overlap with the recent list of similar articles from pubmed. we describe the three selected datasets in detail below. an overview of the datasets is shown in relish. it is a large database in which more than 180k pubmed abstracts were validated in terms of similarity to a seed article @xcite . we utilized the dataset used in the devel-opment of the aspire tool @xcite , which is available for download foot_2 . it contains three levels of similarity and we mapped them as follows: similar (1 or 2), and not similar (0). ## methods when selecting the methods for our experiments, we started with the ones considered in mysore, cohan, and hope (2022), followed by adding some additional ones that we found while researching the literature. we did not consider models trained specifically on one of the datasets, which is the case of aspire @xcite . we describe our methods in this section and we split them into three parts: text processing (cf. 3.1), text representation (cf. 3.2), and ranking (cf. 3.3). ## results we show in this section the results that we obtained with our experiments. for the evaluation, we considered the metrics of precision@20, recall@20, r-precision, and ndcg@20 foot_10 . we considered the top 20 since this is the number of articles that most users usually screen during their search (islamaj @xcite . we summarize the results in two tables, one for an overview of the highest scores (cf. evaluation of the full datasets (eval sets). we first evaluated the full datasets (cf. first columns of ## discussion in this section we discuss some interesting points of our results, such as the best performing methods and a comparison between the datasets, as well as an error analysis. ## conclusions we presented an overview of available methods for the task of text similarity and provided a comprehensive evaluation for three datasets from the biomedical domain. we considered only datasets derived from pubmed and, in our experiments, evaluated various kinds of inputs, such as the title and the abstract of the articles as well as some (potentially relevant) parts of the abstracts, which have been either manually or automatically selected. we provided a variety of evaluations, based on the complete datasets, as well as a modified version of them for a comparison to the ranking algorithm from pubmed similar articles. the results were diverse and the best performing methods were different for each of the datasets, and for the various experiments that we carried out. however, both former and recent methods, e.g., tf-idf and specter, respectively, obtained good results for some of the datasets. further, for all datasets, at least one of our experiments could outperform the ranking algorithm from pubmed similar articles. we hope that our survey can support the researchers when deciding about the most appropriate method to be used for a particular situation, based on either the dataset or hardware availability.