BAGELS : Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text (2025)

Abstract: In scientific research, “limitations” refer to the shortcomings, constraints, or weaknesses of a study. A transparent reporting of such limitations can enhance the quality and reproducibility of research and improve public trust in science. However, authors often underreport limitations in their papers and rely on hedging strategies to meet editorial requirements at the expense of readers’ clarity and confidence. This tendency, combined with the surge in scientific publications, has created a pressing need for automated approaches to extract and generate limitations from scholarly papers. To address this need, we present a full architecture for computational analysis of research limitations. Specifically, we (1) create a dataset of limitations from ACL, NeurIPS, and PeerJ papers by extracting them from the text and supplementing them with external reviews; (2) we propose methods to automatically generate limitations using a novel Retrieval Augmented Generation (RAG) technique; (3) we design a fine-grained evaluation framework for generated limitations, along with a meta-evaluation of these techniques. Code and datasets are available at: Code: https://github.com/IbrahimAlAzhar/BAGELS_Limitation_Gen Dataset: https://huggingface.co/datasets/IbrahimAlAzhar/limitation-generation-dataset-bagels

## introduction in scientific articles, "limitations" refer to the inherent shortcomings, constraints, or weaknesses of a study that may influence its results or restrict the generalizability of its findings @xcite . such limitations can arise from various aspects of the research process, including the methodology, theoretical framework, data collection, experimentation, and analysis @xcite . authors commonly acknowledge issues such as internal validity concerns, measurement errors, confounding factors, and the omission of important variables @xcite . openly discussing limitations is crucial. it upholds credibility and scientific integrity by demonstrating a commitment to ethical and transparent research practices @xcite @xcite . it also clarifies the scope of a study, supporting accurate interpretation, transferability, and reproducibility @xcite . in addition, it helps researchers avoid repeating the same shortcomings @xcite while creating opportunities to refine methods and guide future research @xcite . despite these benefits, researchers are often reluctant to include limitations or articulate them in detail @xcite . concerns about the potential impact on publication chances and career progression @xcite can reinforce this tendency. even when required to acknowledge limitations, as is now common in nlp/ml research, authors sometimes re-sort to generic or irrelevant statements that obscure the study's real constraints @xcite . moreover, limitations may serve as a form of hedging, where findings are presented cautiously to avoid making definitive claims @xcite . this practice, while safer for authors, reduces the clarity and usefulness of the research. failure to disclose limitations undermines the scientific process and misleads readers, reviewers, and policymakers, preventing recognition of constrained findings and potential biases @xcite . meanwhile, the volume of scientific publications has surged @xcite . these factors highlight the need for computational methods to study research limitations. however, progress in nlp toward automatic extraction, generation, and evaluation of limitations remains limited, largely due to the lack of standardized datasets, novel methods, and robust evaluation frameworks. this study takes a step toward closing this gap. our contributions are as follows (see figure 1 ): several studies have examined how limitations are reported in papers. ioannidis (2007) found that only 17% of top-tier articles mentioned limitations, with just 1% doing so in abstracts. similarly, puhan et al. (2012) reported that 27% of biomedical papers lacked limitations, risking overestimation of research reliability. goodman et al. (1994) noted that acknowledging limitations is often problematic in peer review. few journals require discussing limitations (ioannidis, 2007), which can bias reviews and weaken scientific dialogue (horton, 2002), highlighting the need for greater transparency. recent work has explored computational approaches to research limitations. faizullah et al. (2024) proposed an llm-chain pipeline to summarize and refine candidate limitations. al azher et al. (2024) integrated topic modeling with llms to derive structured limitation themes. al azher (2024) developed a graph-augmented llm method for generating detailed limitation statements. other studies address the shortcomings of visualizations by generating more meaningful captions for charts and graphs @xcite . however, these studies are limited to acl/emnlp corpora and rely on author-stated limitations, and use metrics such as rouge and bertscore that miss finer-grained contextual alignment. our framework addresses these shortcomings by including papers from different communities, incorporating peer review comments to capture reviewer perspectives, leveraging cited papers for broader context, and introducing a limitation-level evaluation method to preserve granularity. evaluating nlp outputs is essential for assessing quality, accuracy, and relevance. traditional metrics like rouge and bleu struggle with semantics, while bertscore improves similarity but relies on references and lacks meaningful error analysis. advances in large language models (llms) have opened new evaluation avenues @xcite , from zero-shot and in-context learning @xcite to specialized approaches such as gptscore @xcite , tigerscore @xcite , and pandalm @xcite . other methods include attrscore @xcite , which checks factual support, and summacconv @xcite , which filters low-entailment sentences. despite their promise, llm-based evaluations face issues such as positioning bias, where input order can shift results. we address this by randomizing order and retaining stable outputs. more broadly, our evaluation advances beyond prior work by combining granularity-aware scoring, topic-level agreement, and llms-as-judges. taken together, prior research shows both the need and the opportunity for a more systematic treatment of research limitations. building on these insights, our work unifies dataset construction, limitation generation, and evaluation into a single framework, laying the foundation for more transparent and reproducible analysis of limitations. ## cs graduate students with research experience in nlp and ai the quality of the extraction. the smes from the last step annotated 500 acl papers and 100 neurips papers: one annotator extracted limitations (taking the full section when explicit, or selecting limitation-related sentences when implicit), and two others verified the results. we then compared the tool-based (gpt-4o mini) extractions against this gold standard. notably, the humanextracted (gold) limitations were not segmented; therefore, we combined the llm-extracted limitations and compared them with the gold ones using cosine similarity, precision, recall, f1, and fuzzy matching foot_5 ( ## limitation generation most research papers either do not explicitly mention limitations or underreport them, even when a dedicated section is provided. we compare two systems' ability to generate limitations from research papers: (a) vanilla llm and (b) rag. note that the generators don't have access to the text from where the limitations are extracted, e.g., limitation sections of the papers, paragraphs identified as limitations, or paper reviews; otherwise, the task would be trivial. to improve computational efficiency, we use the three most important sections of a paper as input to the generators rather than the full text. the importance score is computed by the cosine similarity of a section and a reference limitation embedding (see vanilla llm. in the vanilla llm setup, when the input exceeds the context window, it is divided into chunks {p ′ i }, and limitations are generated for each chunk (d' @xcite . the llm is then also asked to aggregate these chunk-specific outputs into a cohesive, meaningful final set of limitations. ## evaluation of generated limitations we want to evaluate the quality of the generated limitations by comparing them with the extracted ones. functionally, both the ground-truth and the predictions are a set of text segments. nlp metrics like bertscore, rouge, and cosine similarity can yield surface overlaps, providing high scores even when the generated limitations are not appropriate, too generic, or imprecise. a possible alternative is to use a holistic llm-as-judge approach, where the generated/ground-truth limitations are merged into single text blocks and then compared. this lacks the point-level granularity needed for fine-grained analysis. we address both these problems by introducing the pointwise (pw) evaluation framework (figure 2 ). problem setup. suppose we have a set of papers @xmath0, p 2 , . . . , p n }. for each paper p i , we assume access to: ground truth limitations g @xmath1, g i2 , . . . , g ix }, where x is the number of ground truth limitations we extracted or annotated for p i . and llm-generated limitations h @xmath2, h i2 , . . . , h iy }, where y is the number of limitations produced by the llm for p i . our goal is to measure (1) how many ground truth limitations the llm correctly reproduces (coverage) and ( 2 ) how well each matched pair of limitations aligns in content and focus (performance). ## experimental setup for generation we use three llms (gpt-3.5, gpt-4o-mini, and llama 3.1 8b 11 ) in a zero-shot setup for both the vanilla generation and rag. the gpt models are accessed through apis, and the llama models are locally deployed with ollama. for the vanilla generation, we also fine-tune three sequence-tosequence models, t5 (512-token window) (raffel et al., 2020), bart (1024 tokens), and pegasus (1024 tokens) @xcite ) on a 70 / 30 train-test split. all models were trained for 3 epochs with a learning rate of 5 × 10 -5 , weight decay 0.01, 300 warmup steps, and batch sizes of 4 (train) and 8 (eval), with early stopping; inputs longer than 512 tokens were truncated. for rag, the vector database is built with llama-index 12 , and the openai text-embedding-ada-002 embedding model for encoding the source and query documents. ## experiments and results evaluation of llm as aligner. the pointwise evaluation protocol above uses an llm to determine whether a generated limitation matches or aligns with a ground truth one. we evaluate gpt 4o-mini's reliability in this task. a set of 100 positive (as per the model prediction) and 100 negative instances is annotated independently by two human evaluators. the human annotators have a cohen's κ score @xmath3, appendix), which shows that the task is largely unambiguous. cohen's κ between human annotators and model prediction is 90-95%, showing exceptional agreement. in comparison, llama-3.1 400b shows poor agreement with the human judges, so in subsequent evaluations, we use gpt-4o mini as the aligner. see ## conclusion we present a new approach for automatically extracting, generating, and evaluating limitations in scientific articles. our method explores incorporating cited works, accommodating top sections of the entire paper, and integrating review feedback to capture perspectives beyond those of the original authors. to evaluate the effectiveness of our system, we introduce a granular text evaluation framework that breaks down limitations into more minor points and employs llms as a judge for assessing alignment. human review validates our extraction and llm-as-judge pipeline, showing strong agreement with expert judgments.