A Zero-Shot Monolingual Dual Stage Information Retrieval System for S panish Biomedical Systematic Literature Reviews (2024)

Abstract: Systematic Reviews (SRs) are foundational in healthcare for synthesising evidence to inform clinical practices. Traditionally skewed towards English-language databases, SRs often exclude significant research in other languages, leading to potential biases. This study addresses this gap by focusing on Spanish, a language notably underrepresented in SRs. We present a foundational zero-shot dual information retrieval (IR) baseline system, integrating traditional retrieval methods with pre-trained language models and cross-attention re-rankers for enhanced accuracy in Spanish biomedical literature retrieval. Utilising the LILACS database, known for its comprehensive coverage of Latin American and Caribbean biomedical literature, we evaluate the approach with three real-life case studies in Spanish SRs. The findings demonstrate the system’s efficacy and underscore the importance of query formulation. This study contributes to the field of IR by promoting language inclusivity and supports the development of more comprehensive and globally representative healthcare guidelines.

## introduction systematic reviews (srs) are crucial in healthcare as they represent the gold standard for synthesising evidence to inform clinical guidelines, policies and practices @xcite . their comprehensive nature ensures that all relevant studies on a particular topic are evaluated, providing a holistic understanding of evidence @xcite . however, existing studies have shown that most srs are skewed towards english databases, excluding databases in languages other than english (loe), even when the srs are geographically specific @xcite ). yet, the results of these studies inform guidelines and practices internationally. additionally, it has been proven that loe papers offer insights, context, or results not available in english-language databases @xcite . these insights alter the direction of association or the effect of meta-analyses @xcite . for instance, a reported sr exploring risk factors for youth violence found that if loe databases were not incorporated, 15-30% of the included studies would have been omitted @xcite . thus, to ensure that srs are comprehensive, which is one of its hallmarks, all evidence must be considered regardless of language or geographical origin. yet, one major problem reported as to why these loe databases are not included in sr is the navigation and search skills required @xcite . these databases often differ from standard english ones, requiring researchers to possess specialised search skills @xcite . thus, valuable research will not be noticed without proper information retrieval (ir) techniques, compromising the comprehensiveness of clinical decision-making. in this study, we aim to use a spanish sr database as a case study. it must be noted that spanish (the fourth most spoken language in the world) is one of the most neglected yet relevant languages used for srs @xcite . we present a stateof-the-art (sota) baseline to serve as a foundation for future research. as such, the present work introduces a foundational benchmark for retrieving spanish biomedical literature and provides insights into, 1) querying the database with search queries obtained from the research question/title as done in english srs database search foot_0 @xcite , and 2) using synonyms of the outcome of the sr as the search strings. for instance, in one of the srs to be used as a case study on mental health salud mental @xcite , we explore the effect of using queries from the research question itself (e.g. ¿cuál es la asociación entre la obesidad o el sobrepeso y los problemas de salud mental entre los niños y adolescentes mexicanos?) or the use of the synonyms outcome of the sr study at hand (depresion y ninos y mexico) provided by human experts. the proposed method consists of a zero-shot dual information retrieval system: a hybrid first-stage retrieval that combines encodings from term-based and a pre-trained language model (plm), and a re-ranker model to further refine the outputs from the first stage. the model is trained with spanish abstracts retrieved from the lilacs database 2 and evaluated on real-life sr case studies. we opted to use lilacs because it stands as the premier and most extensive index for biomedical scientific and technical literature in latin america and the caribbean. existing studies have shown the potential of a two-stage system in ir (retrieval and re-ranking) @xcite @xcite ). yet, these methods are mainly skewed towards english biomedical literature and databases that require labelled or synthetic training data. the main contributions of this paper are 1) compared to existing ir dual-stage retrieval methods skewed towards english biomedical literature, we present a zero-shot dual hybrid first-stage retrieval for spanish sr case studies to serve as a benchmark for future study, 2) we generate and train the dual ir system with the original spanish title as question/query to form the questionabstract pair rather than synthetically generating these questions, considering the particularities of how literature is written in the spanish-speaking world and our available computational resources, and 3) we experiment the model with four sota spanish plm and report and discuss the result as well provide statistical insights into query formation. ## related studies numerous methods have been proposed for ir tasks, including first-stage retrieval techniques, rerankers, and hybrid approaches. to begin, the firststage retrieval has primarily relied on classic termbased probabilistic models like best match (bm25) @xcite , known for their high efficiency and effectiveness, even with very large document collections @xcite . with the advancement in neural networks, dense retrieval methods have been proposed to encode documents and queries calculating relevance 2 https://lilacs.bvsalud.org/es/ through a similarity function @xcite @xcite . recent research in text reranking has increasingly focused on transformerbased models @xcite . @xcite and @xcite proposed using bert-based cross-attention method to capture interactions between queries and passages. other studies have proposed encoder-decoder plm, such as t5, for text binary ranking purposes @xcite @xcite . also, @xcite introduced sgpt for ranking documents. furthermore, @xcite proposed a zero-shot upr plm to re-rank passages directly. recently, studies have focused on hybrid models for ir. authors in @xcite @xcite proposed the use of a combination of bm25 and bert @xcite plm for first-stage retrieval. @xcite presented the use of a sparse traditional bm25 model as the retriever and monobert and duobert as the re-ranker model. @xcite proposed using a hybrid first retrieval with bert and bm25, a cross-attention model trained with pubmed abstract on the bioasq foot_1 test dataset. similarly, @xcite also proposed the use of a zero-shot first-stage model for the bioasq challenge with a dual bert encoder and dual t5-re-ranker model. finally, @xcite presented re2g, a method that combines bm25, gpt-3 and t5 in a dual-stage information system. though these methods have shown promising results, a major setback is all the studies focus on english datasets and some require synthetically generated (document-query pair) for training data which can be computationally costly and affect the authenticity of the data. as such, inspired by the work done by @xcite this study seeks to expand this research frontier by investigating a dual ir retrieval system specifically tailored for spanish biomedical literature in a zero-shot learning environment, addressing the critical. ## results and discussion moving on to ## conclusion and research implication the primary goal of this work was to present a foundational ir model for future research for biomedical spanish systems, marking an initial step towards extending to other understudied languages. additionally, the work sought to provide insights into the impact of query formation on such systems, potentially to guide the development of future query formation tools in such languages. our research findings highlight the potential of hybrid models and re-rankers in enhancing the retrieval of biomedical literature in spanish, thus proposing a potential benchmark for forthcoming spanish ir methodologies. furthermore, our research reveals a notable distinction in query formulation between english and spanish. in english sr retrieval, where queries can be coined from the title/research question and used to retrieve relevant studies, we show that this is different in spanish. in that, the hybrid models or future ir models for spanish literature work may work well with the outcome of the study. as such, querying spanish databases with synonyms of the outcome of the study may help in retrieving relevant information. overall, this re-search contributes to the ongoing efforts to improve the comprehensiveness and precision of literature retrieval in srs, particularly in loe contexts.