Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions (2024)

Abstract: Classification is a core NLP task architecture with many potential applications. While large language models (LLMs) have brought substantial advancements in text generation, their potential for enhancing classification tasks remains underexplored. To address this gap, we propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task. Our extensive experiments and systematic comparisons with various training approaches and a representative selection of LLMs yield new insights into their application for EIC. We investigate the generalizability of these findings on five further classification tasks. To demonstrate the proposed methods and address the data shortage for empirical edit analysis, we use our best-performing EIC model to create Re3-Sci2.0, a new large-scale dataset of 1,780 scientific document revisions with over 94k labeled edits. The quality of the dataset is assessed through human evaluation. The new dataset enables an in-depth empirical study of human editing behavior in academic writing. We make our experimental framework, models and data publicly available.

## introduction generative large language models (llms) have demonstrated substantial advancements in text generation tasks @xcite @xcite . however, their potential for enhancing classification tasks, a significant subset of nlp applications, remains underexplored. the predominant strategy for applying llms to classification tasks is to cast them as generation tasks, followed by instruction tuning @xcite ; (3). provide a first in-depth empirical analysis of human editing behavior using this new dataset. @xcite @xcite , supervised fine-tuning @xcite , and active learning @xcite , all of which aim to generate label strings within the output tokens. recent studies @xcite @xcite have shown the superiority of llms as embedding models on the mteb benchmark @xcite . however, there is a lack of a holistic framework for a systematic study of the classification capabilities of llms in end-toend fine-tuning paradigms. yet, such a framework is important as it extends beyond the current use of llms as generative or embedding models for classification, opens new opportunities for a wide range of real-world tasks, and reveals novel potential for advanced llm training and utilization. to instantiate the framework, we seek a complex, challenging, and underexplored task that is crucial for addressing unresolved real-world applications. edit intent classification (eic) is such a complex task, aiming to identify the purpose of textual changes, necessitating a deep understanding of the fine-grained differences between paired inputs. previous works have provided small humanannotated datasets and demonstrated the crucial role of the intent labels in studying domain-specific human editing behavior @xcite @xcite . however, due to the high cost of human annotation, existing datasets are limited in size. there is a lack of effective nlp automation and extensive labeled datasets to facilitate larger-scale revision analysis. from the modeling perspective, previous studies have primarily explored eic using basic feature engineering @xcite @xcite , fine-tuning small pretrained language models (plms) @xcite , or instruction tuning with llms @xcite . advanced methodologies involving fine-tuning llms remain unexplored. the suboptimal results of previous works ( our work paves the path towards systematically investigating the use of llms for classification tasks. our experiments yield substantial results in the challenging eic task. the resulting large-scale dataset facilitates empirical analysis of human editing behavior in academic publishing and beyond. ## framework we investigate four distinct approaches to fine-tune llms for classification ( §3.1), use various training strategies including three input types ( §3.2) and five transformation functions ( §3.3), systematically comparing different language models ( §3.4). ## application: re3-sci2.0 the original re3-sci dataset contains only 314 documents covering limited research domains, thus constraining in-depth science-of-science analysis of how humans improve scientific quality through revisions and how their document-based editing behavior varies across domains. having determined the optimal approach for eic among the considered ones, we apply our best-performing model to create re3-sci2.0: the first large-scale corpus of academic document revisions for edit analysis across research domains. ## conclusion we have introduced a general framework for finetuning llm classifiers, including four approaches, various llm families, and training strategies. extensive experiments on eic have demonstrated that llms can be effectively fine-tuned as intent classifiers, outperforming fully fine-tuned plms and achieving sota results. among the four approaches, the encoding-based seqc approach has shown superiority in model performance, inference efficiency, and answer inclusion. furthermore, we have demonstrated the versatility of our framework and evaluated the generalizability of our findings on five additional classification tasks. using the best-performing eic model, we have annotated a large-scale dataset of scientific document revisions, enabling in-depth empirical analysis of revision success and human editing behavior across various research domains. our illustratory analysis suggests that (1) focus on clarity and claim modifications and fact/evidence additions significantly and positively impacts revisions success; (2) human editing focus and location remain consistent within the same research domain regardless of document categories but vary substantially across different domains. our work paves the way for systematic investigation of llms for classification tasks and beyond. the general experimental framework is applicable to a wide range of classification tasks. the new dataset provides a robust foundation for multifaceted research in human editing in scientific domain and beyond. the annotation models and the labeling process are reusable and can be applied to generate new high-quality, large-scale automatically labeled revision datasets, as more raw data becomes available.