Transitioning from benchmarks to a real-world case of information-seeking in Scientific Publications (2023)

Abstract: Although recent years have been marked by incredible advances in the whole development process of NLP systems, there are still blind spots in characterizing what is still hampering real-world adoption of models in knowledge-intensive settings. In this paper, we illustrate through a real-world zero-shot text search case for information seeking in scientific papers, the masked phenomena that the current process of measuring performance might not reflect, even when benchmarks are, in appearance, faithfully representative of the task at hand. In addition to experimenting with TREC-COVID and NFCorpus, we provide an industrial, expert-carried/annotated, case of studying vitamin B’s impact on health. We thus discuss the misalignment between solely focusing on single-metric performance as a criterion for model choice and relevancy as a subjective measure for meeting a user’s need.

## introduction scientific publications are one of the primary means by which researchers disseminate their findings and discoveries to the community, but the amount of information to go through can easily become daunting and challenging. exploratory search, i.e, the process of conducting broad and open-ended searches to gain a better understanding of a research topic, is the type of search task that scientists typically spend the most time on. unfortunately, specialized search engines designed for scientists only partially support this type of task, leaving researchers with limited options for efficiently accessing and extracting relevant information from the vast amount of available literature. we are however currently standing at an age of redefining the way we seek information as we make great advances in the whole development cycle of nlp technologies aiming to solve knowledgeintensive tasks. to this end, benchmarks constitute the backbone of this process and fundamentally influence the way we measure progress and identify where future research efforts should be focused. these datasets, almost solely, put the emphasis on performance-driven comparison and create an impression of reliable estimates of progress at a task scale, whereas, in reality, they might not be informative about the way the models would solve human problems or help solve them. in this paper, we illustrate an example of transitioning from two biomedical ir benchmarks, i.e, nfcorpus and trec-covid, to a practical case of seeking information about a specific topic in scientific publications: vitamin b's impact on human health. in this context, we provide an expertannotated collection of relevance judgements on 1811 publications related to vitamin b and health. our goal is to assess how models' comparison on benchmarks is meaningful to solving/assisting the expert seeking such information. we thus show through a zero-shot setting that narrowing the comparison down to a single metric might not be relevant to users' needs, even when a real-world case presents similarities with widely-used benchmarks. our contributions can be summarized as follows: 1. we provide a real-world case of informationseeking in scientific publications that does not drift away from prominent benchmarks' characteristics, 2. we test in a zero-shot setting a few sota models reflecting the current paradigm in nlp and ir and we give an interpretation of their behavior in our case compared to the benchmarks, and 3. we discuss based on our observations the masked phenomena that the current process of evaluation might not reflect. ## experimental setup in this section, we provide a description of our experimental setting of transitioning from information-seeking benchmarks on scientific papers to an industrial exploratory search about vitamin b's impact on health. ## results & discussion we report in nfcorpus bm25 is leading on the three strategies, followed by scientific lms mostly dominating the general robust models. the low scores (compared to the other datasets) can be partially explained with the fact that the percentage of relevant articles is smaller for most queries (≤1-2%). all models, with the exception of bm25 and overall, our results show that models perform differently on datasets and suggest that there is an inconsistency in performance and difficulty in identifying the best model for seeking biomedical information in publications: if starting from nf-corpus, one would suggest using bm25 as a decent model for the vitamin b case, whereas if comparing on trec-covid, one would prefer coco-dr and entirely leave out monot5. in reality, the perfect ndcg@10 on titles of coco-dr might suggest the best fit, but the model is not actually placing all the relevant documents at the smallest ranks: figure 1 illustrates this and shows that the ndcg@10 metric is not reflecting how "early" relevant documents are suggested to the user (the tendency is the same on trec-covid (appendix b) for scincl, coco-dr, and monot5). the differences of scores in ## discussion we further discuss in this section the misalignment between performance measures from the perspective of an expert seeking information in papers. expert search strategy preferences our expert expressed that they sort on titles for more speed, but abstract relevance remains the reference. the reason for this is that titles usually provide information on the study domain as a whole, and can be used to classify into big categories. the abstracts however are used when the title does not allow for immediate classification, since they contain the main question of the paper, the methodology, and the main results. intuitively, models would find more relevance "hints" in abstracts, and thus have greater performance on search strategies that include them. this was rarely the case for all datasets, suggesting that many models might be better at matching shorter contexts (titles being closer to query length compared to abstracts). the success of information-seeking is a process as we previously mentioned, tasks that are complex, such as learning about a new topic, often require multiple search queries and may extend over multiple sessions. it has to be noted that our expert encountered 18 different themes out-of-scope (appendix a) when annotating the entire collection of papers. these themes are discovered during the exploration process, emphasizing the fact that information-seeking is an interactive process and that the reported metric (designed for speed and convenience of ranking systems) is neither informative about the presence of such themes nor about the corresponding response of the different models. as we mentioned in section 2.1, relevance is a subjective measure. our expert investigated the ranked lists returned by different models on the vitamin b use case and categorized the first 100 irrelevant documents for each. we observed that the models' sensitivity to different topics is not the same when measuring similarity. for instance, on titles, coco-dr (best performance) struggled most with practice recommendations, while scincl misjudged the prevalence of b vitamin deficiencies the most. further, this was also the case on abstracts (appendix a) as monot5 struggled most with the vitamin content of food/diet, while bm25 suggested irrelevant studies the most. we illustrate these differences in figure 2 : no agreement whatsoever between models about (ir)relevance of topics, which cannot be reflected by the ndcg@k measure. such a disagreement further complicates the process of identifying the sources of differences, which are important to determine which model may be better suited for specific scenarios, given that such differences might have roots in the training data, model architectures, hyperparameters, or other factors. ## conclusion in this paper, we illustrated the misalignment between single-metric performance and relevancy in practical expert information seeking. through a transition from two biomedical ir benchmarks to a case of an expert seeking information about vitamin b's impact on human health, we showed that the current process of measuring performance may not fully capture the challenges of the task at hand. our observations emphasized the misalignment between relying on top-k ranking metrics and the true nature of the information-seeking process' success. to this end, we provide an extensive description of the use-case creation and relevance judgements to foster future reconciliation between corpus-based evaluations and users' search experience.