Efficient Scientific Full Text Classification: The Case of EICAT Impact Assessments (2025)

Abstract: This study explores strategies for efficiently classifying scientific full texts using both small, BERT-based models and local large language models like Llama-3.1 8B. We focus on developing methods for selecting subsets of input sentences to reduce input size while simultaneously enhancing classification performance. To this end, we compile a novel dataset consisting of full-text scientific papers from the field of invasion biology, specifically addressing the impacts of invasive species. These papers are aligned with publicly available impact assessments created by researchers for the International Union for Conservation of Nature (IUCN). Through extensive experimentation, we demonstrate that various sources like human evidence annotations, LLM-generated annotations or explainability scores can be used to train sentence selection models that improve the performance of both encoder- and decoder-based language models while optimizing efficiency through the reduction in input length, leading to improved results even if compared to models like ModernBERT that are able to handle the complete text as input. Additionally, we find that repeated sampling of shorter inputs proves to be a very effective strategy that, at a slightly increased cost, can further improve classification performance.

## introduction the exponential growth of research publications across various domains @xcite has created an increasing need for automated methods to process scientific texts efficiently. to address this, numerous approaches have been developed to optimize general research workflows, such as literature search @xcite and summarization @xcite . for more specialized tasks, such as extracting specific information from full texts, proprietary large language models (llms) offer potential solutions @xcite . however, these models are not locally deployable, making them expensive to use when processing large datasets. recently, open-source large language models have emerged as strong competitors to proprietary systems, offering comparable performance @xcite . nevertheless, a wider adoption from researchers outside the machine learning research community is unlikely within the next years, primarily due to their significant hardware requirements. furthermore, both proprietary and open-source llms of this scale are highly energy-intensive, raising concerns about their sustainability. this highlights the importance of exploring smaller, more efficient models that can deliver similar performance while minimizing resource consumption, or of exploring other strategies to reduce the computational cost of using these models to solve specific tasks. to address these challenges, we investigate the potential of more efficient bert-based models alongside slightly more resource-intensive local large language models (llms) for classification of scientific full texts. as part of this effort, we introduce the eicat dataset, which consists of scientific full-text papers focused on specific invasive species and their impact on the native ecosystem, as well as labels specifying the impact category of that species with corresponding evidence sentences that were extracted from the papers. in our series of experiments, we first evaluate the performance of a standard bert-based classifier on the eicat dataset, where full-text inputs must be split into multiple segments due to the limited context length. we then compare its performance to modernbert @xcite , a recent bert variant capable of handling longer contexts, as well as to llama-3.1 8b @xcite , a state-of-the-art local llm. all models face significant challenges due to the large input size, leading us to performing further experiments with training selector models to identify and prioritize the most relevant input sentences for training and evaluation. to ensure the general applicability of this approach, we test various sentence selection strategies, including leveraging human-provided evidence annotations, using llm-generated selections, and using importance scores extracted from the classifiers. our findings indicate that many selection strategies improve classifier performance while simultaneously enhancing the efficiency of the decoder model. for scenarios where efficiency is less critical, we also observe that repeated randomization can improve the classification performance and even make a random selection of input sentences a viable strategy, thus leading to a simple-to-use way of boosting classification results. ultimately, this work presents a generalizable pipeline for accelerating inference and improving performance of scientific full-text classification. the remainder of the paper is organized as follows: section 2 reviews recent natural language processing approaches for the automated processing of scientific texts. section 3 introduces the eicat dataset, while sections 4 and 5 describe our experiments and present the results. section 6 provides a discussion of the findings, and section 7 concludes with final remarks. ## the eicat dataset we present a new dataset for training and evaluating models on the task of assessing the impact of invasive species on ecosystems based on scientific full texts. this dataset is grounded in the "environmental impact classification for alien taxa" (eicat, iucn (2020)) standard, a classification standard developed by the international union for conservation of nature (iucn), which is used by researchers to compile standardized summaries of scientific literature addressing invasive species, along with assessments of the species' impacts as reported in these publications. the impacts are categorized into one of six possible classes: minimal concern, minor, moderate, major risk, massive, and data deficient. furthermore, researchers extract and include sentences from the full texts as evidence supporting their selected category. these impact assessments for various species are publicly available as excel files at https://www.iucngisd.org/gisd/ . to construct our dataset, we acquired impact assessment files for as many species as possible. from these files, we extracted publication names and corresponding impact assessments for each species, covering around 800 publications. using llama-3 8b, we determined whether each citation represented a scientific paper (as opposed to books, phd theses, government reports, etc.), since this study focuses exclusively on shorter scientific articles. we then used crossref (crossref.org) as well as manual scraping to obtain as many full texts as possible. since the retrieved documents were in pdf format, we used grobid @xcite to extract the raw text from the publications. we excluded any documents for which text extraction was unsuccessful, resulting in a final dataset with 436 full texts addressing 120 species. as a final processing step, we matched the evidence sentences from the impact assessments to sentences in the extracted text files. discrepancies between the version of the paper we obtained and the one used for the assessments, as well as artifacts introduced during the pdf-to-text conversion, made exact matching infeasible in many cases. to address this, we implemented a fuzzy matching strategy, matching two sentences if they contain most of the same words in the same order. for matches slightly below the set similarity threshold, we used llama-3 to determine whether the sentences were still a valid match. in total, we identified 2,247 evidence sentences, compared to 2,226 sentences in the original annotations. the higher count likely results from imperfect matching, as well as from pdf-to-text conversion artifacts, which sometimes split evidence sentences in the full text into two parts. we created training, validation, and test splits comprising 82%, 8%, and 10% of the species, respectively. to prevent inflated performance scores caused by the model learning the typical impact category assigned to a specific species across publications, all texts addressing the same species were assigned to the same split. we publish the dataset containing publication names, impact labels and evidence sentences together with our code on github.com/inasargumentation/efficient full text classification. ## discussion in our evaluation, we identified significant challenges when using instruction-tuned llms for scientific text processing. on the one hand, extracting a different set of sentences, even if they should contain the necessary information for performing the classification, can easily change classification results and even push the model towards incorrect conclusions. additionally, a detailed natural language description of our task was insufficient for the llama model to achieve results comparable to a 70 times smaller bert classifier, and the explicit selection of highly relevant sentences through the evidence selector did not yield improvements. we interpret this as a sign that sample-level labels, as used by bert, provide substantially more information than both evidence annotations and natural language descriptions. the superior informational content of samplelevel labels compared to evidence annotations is plausible considering that a single evidence annotation only conveys information about a single sentence, while a sample-level label provides information about every sentence in the whole text. on the other hand, the superior performance of sample-level labels over natural language descriptions is especially significant given the recent trend toward prompting-based approaches rather than extensive labeling efforts. in-context learning @xcite offers a potential bridge between these approaches, enabling the delivery of rich sample-level information to llms without training, typically complementing natural language descriptions within the prompting framework. this combination can thus potentially overcome the challenge of precisely specifying a given task by returning to the classical way of demonstrating desired behavior. however, while this approach has shown success, it becomes impractical for tasks involving lengthy inputs, such as scientific full-text classification. for such cases, finetuning local large language models could present a viable solution, which can be explored in future work. ## conclusion we introduced a novel dataset for scientific fulltext classification and conducted extensive experiments using smaller encoder and larger decoder architectures. our results demonstrated that various strategies for reducing input size can simultaneously enhance efficiency and performance, offering a generalizable pipeline adaptable to other tasks. however, as classification scores remain suboptimal, future research could investigate the potential of fine-tuning local llms, leveraging recently emerging llms with advanced reasoning capabilities @xcite , or testing the performance of larger proprietary models.