LGAR : Zero-Shot LLM -Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews (2025)

Abstract: The scientific literature is growing rapidly, making it hard to keep track of the state-of-the-art. Systematic literature reviews (SLRs) aim to identify and evaluate all relevant papers on a topic. After retrieving a set of candidate papers, the abstract screening phase determines initial relevance. To date, abstract screening methods using large language models (LLMs) focus on binary classification settings; existing question answering (QA) based ranking approaches suffer from error propagation. LLMs offer a unique opportunity to evaluate the SLR’s inclusion and exclusion criteria, yet, existing benchmarks do not provide them exhaustively. We manually extract these criteria as well as research questions for 57 SLRs, mostly in the medical domain, enabling principled comparisons between approaches. Moreover, we propose LGAR, a zero-shot LLM Guided Abstract Ranker composed of an LLM based graded relevance scorer and a dense re-ranker. Our extensive experiments show that LGAR outperforms existing QA-based methods by 5-10 pp. in mean average precision. Our code and data is publicly available.

## introduction the scientific literature grows rapidly, by an estimated average of roughly 7% per year over the past 22 years @xcite , with 3.3 million articles having appeared worldwide in 2022 @xcite . keeping track of the state-of-the-art of a particular research topic is hence inherently difficult. systematic literature reviews (slrs) are an academic standard of investigating the current stateof-the-art in a specific field. they follow a strict protocol (e.g., @xcite ). since conducting slrs is typically very expensive and time-consuming @xcite , efforts have been made to automate various parts of the process. most existing automation figure 1: lgar method for abstract screening consisting of a 2-stage ranking approach: (1) the llm ranker makes use of a realistic specification of the slr to assign high-level graded relevance scores, (2) dense re-rankers produce a fully ordered list in scalable time. due to the difficulty of leveraging llms for ranking in a scalable manner @xcite , almost all prior work on abstract screening with llms works in a binary classification setting, refraining from inducing a ranking of candidate papers. in practice, a ranked list is much more useful, because the decision of the cut-off point can be left to the human user flexibly. the only existing llmbased ranker (akinseloyin et al., 2024) 2 generates questions for slr criteria and generates answers for each abstract. the answers are converted to numeric scores using a sentiment classifier, extended with embedding-based similarities, and averaged. while promising, this pipeline-based approach suffers from error propagation in the question generation and sentiment analysis steps. existing abstract screening datasets @xcite provide only part of the slr criteria. none of them provides the research questions underlying the slrs. prior ranking studies approximate the information need by using the slr's title and boolean document retrieval queries (e.g., @xcite , or use the incomplete set of criteria specified in the slr abstracts @xcite . to evaluate llm-based rankers using the same slr descriptions as humans do, we manually collect criteria and research questions exhaustively from the full texts of 57 slrs and validate our collection process. this important dataset extension will be made publicly available to foster future work. we propose lgar, a novel zero-shot llm-guided abstract ranker, which directly leverages llms to provide graded relevance judgments. a dense ranker @xcite then re-ranks the sets of abstracts with the same score. our experiments using our new complete set of criteria and recent open-weights models for all baselines show that lgar outperforms the existing state-ofthe-art system @xcite by a large margin (5-10pp. in mean average precision). our contributions are as follows: (1) we provide the exhaustive and validated set of inclusion and exclusion criteria, as well as research questions for two slr datasets, enabling realistic experiments. (2) we propose lgar, a zero-shot system for llm-based abstract screening that performs robustly across datasets and llm families. (3) our extensive experimental evaluation highlights the superior performance of lgar compared to existing zero-shot and fine-tuned methods. we perform a detailed ablation study for the task of abstract ranking with llms. we find that using large scales with up to 20 options is most effective. this differs from general ranking tasks, where scales with up to five choices have been reported to be optimal. 2 the approach was also recently tested on four slrs on the impact of climate change on diseases @xcite . ## background and related work this section provides a brief overview of slr automation with a focus on abstract screening. slr automation. in the first stage of an slr a set of candidate papers is retrieved from scientific databases, typically using boolean queries. the automatic formulation of these queries based on review protocol text has been approached using semantic parsing @xcite , rank fusion @xcite , and llm-based generation @xcite . slrs have also been supported by applying summarization methods @xcite and by extracting fine-grained entity types and relations @xcite . most active learning (al) tools for slr automation use traditional machine learning (ml), e.g., as review (van @xcite , abstrackr @xcite , @xcite , fastread @xcite , rayyan @xcite , or robotanalyst @xcite . @xcite compare neural classifiers in an al setup. al assumes substantial training data. in contrast, we focus on zero-shot settings. abstract screening. early approaches for binary relevance classification for abstract screening apply traditional ml techniques @xcite @xcite . @xcite fine-tune several bert models @xcite as binary classifiers and combine their predictions. first learning-to-rank setups for abstract screening use non-neural techniques @xcite . @xcite combines sentence-level relevance scores computed by a convolutional neural network. @xcite use a cross-encoder approach following monobert @xcite . the first abstract screening methods using llms all operate in classification settings, applying chatgpt in a zero-shot setting @xcite . @xcite also fine-tune llama @xcite and guanaco @xcite to generate binary inclusion labels. @xcite compare a larger set of open-weights llms on a calibrated binary classification task. ranking with llms. integrating llms with ranking approaches is a highly active research area that has just recently emerged @xcite . llms can be integrated as query rewriters, as dense retrievers, as "readers" to generate natural language answers, or for data augmentation. we here focus on their application in the ranking stage, where they can be applied using a pointwise @xcite , pairwise @xcite , or listwise approach @xcite . the latter two are inefficient for long candidate lists, which is usually the case in abstract screening. in pointwise ranking, the generation likelihood of the query given the document can be used as relevance score @xcite , or the llm can be prompted to generate a relevance score @xcite . @xcite show that adding intermediate relevance options can improve ranking performance over binary setups. as abstract screening queries are long and complex, we turn to relevance generation. to the best of our knowledge, the only existing ranking approach for abstract screening using llms (akinseloyin et al., 2024) uses a qa setup. for each inclusion/exclusion criterion, a question is generated using chatgpt, and the llm is prompted to answer them for each paper. answers are converted to numerical scores based on whether a bart @xcite ) model trained on sentiment analysis assigns positive (1), neutral (0.5), or negative (0). the final score for each paper is computed by averaging these scores with embedding similarities of the questions, the original criteria, and the abstract. akinseloyin et al. (2024) report that while promising, their pipelinebased approach suffers from error propagation. our own system, lgar, uses a more effective combination of zero-shot llm-based pointwise relevance generation and monot5-based re-ranking. ## dataset extensions in this section, we describe our extension of two abstract screening benchmarks with inclusion/exclusion criteria and research questions to facilitate more realistic evaluation setups. we extend the synergy dataset @xcite and the test set of the clef technological assisted re-view 2019 (tar2019) dataset (task 2) @xcite . 3 both datasets are in english and provide the title of each slr and a set of papers labeled with binary relevance judgments. 4 ## llm-guided abstract ranking we propose lgar, a ranking model comprised of a two-stage process to determine the relevance of a screened paper, as illustrated in figure 1 . in the first stage, the llm assigns each paper a relevance score on a scale from 0 to k, similar to the llmbased ranking approach direct(0,k) of @xcite . ranking for abstract screening requires producing an ordered list of potentially thousands of papers. the differentiation that llms are able to produce is limited to the range of a dozen relevance scores @xcite , i.e., the first ranking step will produce many ties. we hence produce the global ranking by additionally ranking the papers within each set of papers that have obtained the same relevance score by the llm using a zero-shot dense ranking model. our proposed two-stage ranker is modular, providing for modular combinations of llms and dense rankers. ## experiments to compare lgar to existing approaches, we conduct an extensive experimental evaluation using our newly constructed benchmark. in this section, we describe our experimental design, including our selection of models, our choice of evaluation metrics, and the baselines. we then present our experimental results, followed by several ablation studies. ## discussion and outlook our results suggest that llms are very effective in interpreting inclusion and exclusion criteria for abstract screening. they provide a good initial categorization. re-ranking is particularly effective for finding relevant papers accidentally categorized as irrelevant by the llm, thus significantly contributing to workload savings. compared to the approach of akinseloyin et al. (2024), it does not seem necessary to split the criteria into several parts, which significantly reduces the number of llm requests. at the same time, lgar is less "explainable" than models evaluating every criterion separately. we hypothesize that lgar performs more robustly precisely because it is not forced to evaluate each criterion separately, which likely introduces many noisy scores in the qa-based model. for many criteria, the abstract may not contain sufficient information to allow a meaningful assessment. the qa-based baseline is supposed to generate neutral in such cases, yet, detecting when information is insufficient to answer a question is yet a difficult task for llms @xcite . in future work, it would be interesting to further support the human user by identifying the criteria that led to including or excluding a paper. ## conclusion in this paper, we have proposed lgar, a novel llm-guided abstract ranker for slrs. lgar is based entirely on zero-shot open-weights models, yet outperforms models specifically tuned for this task. we contribute a dataset extension of exhaustively extracted criteria and research questions and compare to baselines in a fair way, using the same backbones and inputs, finding that lgar performs best on two abstract screening datasets.