Measuring the relative importance of full text sections for information retrieval from scientific literature. (2021)

Abstract: With the growing availability of full-text articles, integrating abstracts and full texts of documents into a unified representation is essential for comprehensive search of scientific literature. However, previous studies have shown that naïvely merging abstracts with full texts of articles does not consistently yield better performance. Balancing the contribution of query terms appearing in the abstract and in sections of different importance in full text articles remains a challenge both with traditional bag-of-words IR approaches and for neural retrieval methods. In this work we establish the connection between the BM25 score of a query term appearing in a section of a full text document and the probability of that document being clicked or identified as relevant. Probability is computed using Pool Adjacent Violators (PAV), an isotonic regression algorithm, providing a maximum likelihood estimate based on the observed data. Using this probabilistic transformation of BM25 scores we show an improved performance on the PubMed Click dataset developed and presented in this study, as well as the 2007 TREC Genomics collection.

## introduction pubmed ( https://pubmed.gov ) is a search engine providing access to a collection of more than 30 million biomedical abstracts. of these, about 5 million have full text available in pubmed central (pmc; https://www.ncbi.nlm.nih.gov/pmc ). millions of users search pubmed and pmc daily @xcite . however, it is not currently possible for a user to simultaneously query the contents of both databases with a single integrated search. with the growing availability of full-text articles, integrating these two rich resources to allow a unified retrieval becomes an essential goal, which has potential for improving information retrieval and the user search experience @xcite ). an obvious benefit is improving the handling of queries that produce limited or no retrieval in pubmed. in many instances, incorporating full text information can yield useful retrieval results. for example, the query cd40 fmf retrieves no articles in pubmed, but finds 60 articles in pmc discussing protein cd40 and a computational technique of flow microfluorometry @xcite . a number of studies have pointed out the benefits of full text for a range of text mining tasks @xcite @xcite and demonstrated improved performance on named entity recognition, relation extraction, and other natural language processing tasks @xcite . for information retrieval, however, combining the full text of some papers with only the abstracts of others is not a trivial endeavor. naïvely merging the body text of articles with abstract data, naturally increases the recall, but at a cost in precision, generally degrading the overall quality of the combined search (w. @xcite . this can be explained by several complexities associated with full texts, such as multiple subtopics often being discussed in a fulllength article or information being mentioned in the form of conjecture or a proposal for future work. in addition, not every record matching the query is focused on the query subject, as query words may be mentioned in passing, which is more common in full text. another challenge in incorporating full text in retrieval is merging sources of information with different characteristics: the abstract, generally a concise summary on the topic of the study, versus a lengthy detailed description provided in full text. to address that, recent studies have attempted to use full text in a more targeted way -by performing paragraph-level retrieval @xcite , passage-level retrieval @xcite or sentencelevel retrieval @xcite measuring the relative importance of full text sections for information retrieval from scientific literature. ## evaluation datasets retrieval methods are generally evaluated based on how the retrieval output compares to a gold standard. a gold standard is a set of records judged for relevance to a query that provides a benchmark against which to measure the quality of search results. this approach is used at the annual text retrieval conference (trec), run by the national institute of standards and technology (nist) @xcite . nist develops a list of queries, called topics, and provides large test collections and uniform scoring procedures. the difficulty with this approach is that a gold standard is created by human experts which makes the evaluation expensive, time consuming, and therefore not available for large scale experiments involving thousands of queries. to compare different retrieval approaches without a manually created gold standard we describe semi-automatically created test data based on indirect human judgements that can be utilized in our setting. the pubmed user click dataset is created based on retrospective analysis of pubmed queries under the assumption that a clicked document is relevant to a user issuing a query. in our study we use both, the trec 2007 genomics and pubmed user click datasets. trec 2007 genomics dataset. the genomics dataset @xcite consists of 36 queries, called topics, and 162,259 full-text articles from highwire press ( http:// highwire.stanford.edu/ ). 160k of these documents were successfully mapped to their corresponding pubmed identifiers (pmids) and are the basis of our experiments. each document is split into legal spans corresponding to paragraphs in the articles, amounting to over 12 million legal spans. for each of the 36 topics human relevance judgements are provided on the paragraph level. following previous studies, a document is labeled positive, if it contains at least 1 paragraph judged to be relevant to the query. the query topics are presented in the form of biological questions, such as: what toxicities are associated with etidronate? what signs or symptoms are caused by human parvovirus infection? these question-like topic formulations contain generic words, that are not representative of the specific information need of a user, such as "what", "associated", etc. we applied a combination of frequency-based techniques and manual validation to filter these stop words out and used the remaining 165 content terms for our analysis. ## methods -using full text to score a query token here we examine how to optimally use bm25 scores coming from the abstracts and full text paragraphs to improve retrieval performance. we first define the score of a token within a full-text section, which then we transform into a probability of that document being relevant given the score and the section. we then learn how to combine these section-based token scores into an overall score predicting the probability of a document being relevant. ## results proposed methods are tested on the pubmed click dataset and on the trec genomics collection @xcite . ## conclusions and discussion based on the pubmed click dataset and the trec genomics dataset, we studied how to integrate full text and abstract information for scoring a query token. the main contribution of this work is to study the benefits of log odds of bm25 compared to raw bm25 scores. our experimental results on both datasets support these important conclusions: for the pubmed click dataset, using the sum logodds score from the whole document for a query token produces better results than using only the abstract score. in the trec genomics dataset, the performance of full text logodds is comparable to abstract only score. this is an important contribution and meaningful building block towards improving full text retrieval in pubmed. our immediate plan is to extend this single token analysis to full queries.