Predicting The Scholarly Impact of Research Papers Using Retrieval-Augmented LLM s (2025)

Abstract: Assessing a research paper’s scholarly impact is an important phase in the scientific research process; however, metrics typically take some time after publication to accurately capture the impact. Our study examines how Large Language Models (LLMs) can predict scholarly impact accurately. We utilize Retrieval-Augmented Generation (RAG) to examine the degree to which the LLM performance improves compared to zero-shot prompting. Results show that LLama3-8b with RAG achieved the best overall performance, while Gemma-7b benefited the most from RAG, exhibiting the most significant reduction in Mean Absolute Error (MAE). Our findings suggest that retrieval-augmented LLMs offer a promising approach for early research evaluation. Our code and dataset for this project are publicly available.

## introduction evaluating the impact of a research paper is important to the scientific process, as researchers, funding agencies, and policymakers must make informed decisions @xcite . typically, impact has been measured using bibliometric indicators, such as citation counts, h-index, i-index, and journal impact factors @xcite , as well as field-normalized metrics such as field citation ratio (fcr) and relative citation ratio (rcr) @xcite . while each of these metrics provides useful insights into a paper's impact @xcite , they depend on citation data, which takes time to accumulate. the delay in assessing the scholarly impact can cause a challenge when making decisions in certain situations. for example, organizations that allocate funding for investments may need to assess the 1 code 2 dataset potential of new publications to guide funding, or domains that are evolving quickly may need to identify influential work that is important for directing researchers' attention. while alternative metrics such as altmetrics attempt to capture the engagement of the public immediately through social media and news coverage @xcite @xcite , they also rely on data after publication and thus cannot provide a true preemptive evaluation. in this study, we estimate the scholarly impact of research papers by analyzing their content using large language models (llms). llms have opened new possibilities for evaluating impact @xcite , allowing researchers to rigorously analyze the research paper is content for more insights @xcite @xcite . however, despite these models being trained on a vast corpus, their knowledge is fixed at the time of training, so they can't dynamically access external sources during inference @xcite . this limitation means that for predicting scholarly impact, llms cannot evaluate how a new paper compares to prior related studies or assess its contribution in the context of ongoing research. since a paper's influence often depends on how original it is compared to prior work and how relevant it is to ongoing research, @xcite , comparing it to other studies is essential for accurately assessing its potential impact. to address this concern, we use a technique called retrieval augmented generation (rag), where the retriever collects external sources that are semantically similar to the query being evaluated. these sources are sent to the llm as context to give a more informed response @xcite . in the context of scientific articles, recent work shows that rag improves the generation of structured scientific content, such as future work statements, by grounding predictions in relevant prior research @xcite . rag could potentially be valuable in the case of impact prediction, where we use prior literature to help llm reason better. in our study, we will be addressing the following research questions: rq1: does rag improve the overall performance of llm compared to zero-shot for scholarly impact prediction? rq2: how well do predictions from llms generalize across different research disciplines? rq3: how often did rag improve or degrade llm performance among individual papers? ## dataset collection we collected research articles published between 2018 and 2022 across five disciplines: computer science, mathematics, engineering, physical sciences, and psychology. for each discipline and year, we randomly sampled 2,000 articles, extracting their titles, abstracts, and fcr scores from dimensions.ai foot_0 . this creates a diverse dataset, sufficient to test the generalizability of prediction models. the fcr adjusts a paper's citation count by comparing it to the average citations of papers in the same field and publication year @xcite . since fcr is an unbounded metric with no upper limit, we used the empirical cumulative distribution function (ecdf) to normalize its values within a 0 -1 range for each discipline and publication year. this makes our data more consistent and suitable for the model to learn and analyze for prediction @xcite . we then preprocessed the title and abstract columns by converting text to lowercase, discarding special characters, and removing abstracts with fewer than 100 tokens. based on the ecdfnormalized fcr values, each article was categorized into one of three impact levels: low (0-0.33), medium (0.34 -0.66), or high (0.67 -1) impact level. these categorical labels were used to examine the distribution of impact levels within the dataset. to mitigate class imbalance and reduce the risk of model overfitting or bias toward dominant citation patterns, we removed overrepresented classes. we then evaluated the readability of each paper's abstract using the textstat foot_1 library. these readability metrics were used as additional input features to quantify the ease or difficulty of text comprehension. since readability can influence citation patterns, including these metrics enables us to investigate its potential role in the llm's ability to predict scholarly impact @xcite . we used the flesch reading ease (fre) score foot_2 , where higher values indicate more readable text, and the gunning fog index (gfi) foot_3 , which estimates the years of education required to comprehend the text @xcite . after preprocessing, the dataset consisted of 6, 000 research articles, each containing its title, abstract, abstract readability scores, and normalized fcr scores. in the experiment, we divided the dataset into a knowledge base containing 5, 400 papers (90% used for retrieval) and a test set of 600 papers (10% used to evaluate the model's predictions). ## methodology figure 1 illustrates our workflow, which uses rag to assist the llm in making its prediction. the experiment used three llms (llama3-8b, mistral-7b, and gemma-7b) and a retrieval-augmented setup that combined dense retrieval for contextual grounding and self-consistency to improve prediction reliability. besides rag, we used zero-shot as a baseline to assess how much the retriever actually benefited the llm performance. ## experimental setup we evaluated the llm performance in both the zero-shot (figure 2 , appendix) and rag (figure 3 , appendix) using two types of input sets: (1) text only (title, abstract) and (2) text with readability (title, abstract, flesch reading ease, gunning fog index). we downloaded each model from ollama foot_5 to run locally with the default configurations. we implemented a dense retrieval approach using faiss to identify the most relevant research papers from the knowledge base. the faiss index was created using the indexflatip method, which is well-suited for cosine similarity search when used with normalized embeddings. since we used scibert to generate the embeddings, we normalized each vector before indexing to ensure that the inner product search approximated the cosine similarity. to efficiently compute embeddings, we processed research papers in batches of 1, 000, using parallel execution with four workers to speed up the computation. the resulting embeddings were stored directly in faiss, enabling a flat, brute-force retrieval strategy. during the retrieval phase, the title and abstract of each input paper were encoded using scibert and used to query the faiss index. the retriever will then find five papers that are the most semantically similar to the query containing the test paper and pass them to the llm, where it will then use those five papers as context when predicting the normalized fcr score. to evaluate the performance of the llms, we measured accuracy and ranking quality using mean absolute error (mae) and normalized discounted cumulative gain (ndcg). mae quantifies accuracy by calculating the average difference between predicted and actual impact scores, with lower values indicating higher accuracy. ndcg assesses how well the model ranks papers by impact, comparing its predicted rankings of fcr scores to their actual rankings, where a value closer to 1 means that it is more accurate at ranking high-impact papers. rq2: llm prediction generalizable across domains. the influence of rag on the accuracy of the prediction varied across different domains (figure 4 , appendix), with some fields benefiting more than others. computer science and engineering showed the most significant improvements across most models, with gemma-7b showing a reduction in mae of 0.105 in both fields, the most substantial gain among all domains. mistral also showed strong improvements, decreasing its mae by 0.055 in computer science and 0.059 in engineering, while llama3-8b showed the highest improvement in engineering only. rq3: how often rag improve or degrade llm performance. to assess whether the retriever improved or worsened the llm performance, we compared the absolute prediction error of rag and zero-shot for each paper across all llms. overall, rag achieved a performance superior to zero-shot in 57 -59% in all cases (mistral-7b: 1, 369, llama3-8b: 1, 376, gemma-7b: 1, 409), while zero-shot outperformed rag in 36 -38% of cases (mistral-7b: 873, llama3-8b: 908, gemma-7b: 874). this further shows that the context provided by the retriever generally improved the prediction but was not universally effective. these results reveal that while rag can help, it also introduces noise or conflicting information, a challenge also addressed in astute rag, which investigates how to detect and mitigate such retrieval failures in llms @xcite . ## conclusion the evaluation process provides insight into a paper's impact and contribution to the research community. our study attempts to expedite that process by prompting an llm with a paper's title, abstract, and abstract readability. to improve the llm response, we also incorporate rag, which retrieves relevant papers as context when llm makes its prediction, offering a faster alternative for assessing impact. while rag improved the prediction overall, its inconsistent performance in some instances highlights the need to refine the retrieval approach further.