Multi-label Sequential Sentence Classification via Large Language Model (2024)

Abstract: Sequential sentence classification (SSC) in scientific publications is crucial for supporting downstream tasks such as fine-grained information retrieval and extractive summarization. However, current SSC methods are constrained by model size, sequence length, and single-label setting. To address these limitations, this paper proposes LLM-SSC, a large language model (LLM)-based framework for both single- and multi-label SSC tasks. Unlike previous approaches that employ small- or medium-sized language models, the proposed framework utilizes LLMs to generate SSC labels through designed prompts, which enhance task understanding by incorporating demonstrations and a query to describe the prediction target. We also present a multi-label contrastive learning loss with auto-weighting scheme, enabling the multi-label classification task. To support our multi-label SSC analysis, we introduce and release a new dataset, biorc800, which mainly contains unstructured abstracts in the biomedical domain with manual annotations. Experiments demonstrate LLM-SSC’s strong performance in SSC under both in-context learning and task-specific tuning settings. We release biorc800 and our code at: https://github.com/ScienceNLP-Lab/LLM-SSC.

## introduction with the increasing number of published scientific papers today, researchers face significant challenges in quickly pinpointing needed information. to address this problem, organizing complex paper content according to the rhetorical roles of each sentence in a structured format has garnered interest @xcite @xcite . since the rhetorical roles of each sentence are often informed by the context from neighboring sentences (e.g. appendix d), * corresponding author this task is referred to as sequential sentence classification (ssc) @xcite . ssc can enable the fine-grained information retrieval @xcite , enhance extractive summarization @xcite , and improve other downstream tasks. for example, labeling objective sentences in scientific abstracts can support information retrieval based solely on the papers' objectives. existing studies have explored ssc using contextualized language representations. artificial neural network (ann)-based ssc methods typically follow a hierarchical structure: an encoding layer to represent word tokens and embed sentences, a context interaction layer to enhance sentence embedding with surrounding context, and a labeling optimization layer to produce optimized sequential labels @xcite @xcite @xcite @xcite . other research utilizes the masked token objective or transformers, introducing special tokens to encode contextual information and using these tokens to predict sequential labels @xcite . despite promising progress in the ssc task, several gaps remain, including pretrained model size, input sequence length, multi-label annotation, and dataset creation. specifically, current ann-and transformer-based methods have only employed moderately sized pretrained models (e.g., word2vec @xcite , scibert @xcite ), while the application of large language models (llms) in ssc is underexplored. furthermore, the existing transformerbased methods rely on bert, which is constrained by 512-token sequence length limit @xcite . additionally, ssc has not expanded from single-label to multi-label, which is essential since a sentence can serve multiple rhetorical roles within a context @xcite . moreover, the widely used ssc dataset in the biomedical field, pubmed 200k rct, is automatically generated from structured abstracts in pubmed @xcite . however, this dataset does not include unstructured abstracts with free-form writing styles, which may deviate from the common patterns found in structured abstracts @xcite . to bridge these gaps, this paper explores the application of llms in multi-label ssc using manually created datasets. we propose llm-ssc, a novel unified framework for in-context learning and parameter-efficient finetuning (peft) using gemma-2b @xcite for this task. unlike existing approaches that create contextual representations of sequential sentences, llm-ssc leverages llms to generate ssc labeling results based on designed prompts, which include a demonstration part to showcase the task and a query part to introduce the prediction target. to address the challenge of multi-label annotation, we design an auto-weighting multi-label contrastive learning loss that relaxes the constraint of formation of positive and negative pairs in the contrastive learning and reweights the importance of positive and negative pairs based on their label information. our contributions are as follows: ## methods in this section, we first introduce the notation and then present llm-ssc, an llm-based frame-work for sequential sentences in-context learning and parameter-efficient finetuning, integrating complete contextual information within the prompt and consider neighboring context when making predictions. to enable the multi-label classification, we propose auto-weighting multi-label contrastive learning loss. the overview of the proposed framework is shown in figure 1 . ## conclusion in this paper, we introduce llm-ssc, a unified framework for in-context learning and parameterefficient llm finetuning for multi-label sequential sentence classification problem. llm-ssc integrates complete contextual information within the prompt and considers neighboring context when making predictions. additionally, we present a multi-label contrastive learning loss with autoweighting scheme to reweight the importance of negative pairs and address the multi-label sequential sentence classification problem. furthermore, we release biorc800, a manually annotated multi-label ssc dataset using unstructured abstracts from the biomedical field, contributing to the development of more robust methodologies for this task. extensive experiments validate the re-markable capability of llm-ssc in ssc tasks under both in-context learning and parameter-efficient finetuning settings.