S ci P rompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics (2024)

Abstract: Prompt-based fine-tuning has become an essential method for eliciting information encoded in pre-trained language models for a variety of tasks, including text classification. For multi-class classification tasks, prompt-based fine-tuning under low-resource scenarios has resulted in performance levels comparable to those of fully fine-tuning methods. Previous studies have used crafted prompt templates and verbalizers, mapping from the label terms space to the class space, to solve the classification problem as a masked language modeling task. However, cross-domain and fine-grained prompt-based fine-tuning with an automatically enriched verbalizer remains unexplored, mainly due to the difficulty and costs of manually selecting domain label terms for the verbalizer, which requires humans with domain expertise. To address this challenge, we introduce SciPrompt, a framework designed to automatically retrieve scientific topic-related terms for low-resource text classification tasks. To this end, we select semantically correlated and domain-specific label terms within the context of scientific literature for verbalizer augmentation. Furthermore, we propose a new verbalization strategy that uses correlation scores as additional weights to enhance the prediction performance of the language model during model tuning. Our method outperforms state-of-the-art, prompt-based fine-tuning methods on scientific text classification tasks under few and zero-shot settings, especially in classifying fine-grained and emerging scientific topics.

## introduction scientific text classification tasks involve categorizing scientific abstracts into specific disciplines or topics. recent studies leverage prompt-based finetuning method (ding et al., 2022a; gu et al., 2022; 1 our code is available at https://github.com/ zhiwenyou103/sciprompt . @xcite , transferring the text classification problem as a masked language modeling task. masked language models (mlms) are developed by extensively training on large text corpora with a percentage of the input tokens being randomly replaced with a [mask] token. traditional fine-tuning, which requires additional training on labeled domain-or task-specific data @xcite , may not be suitable in limited data scenarios, such as few and zero-shot settings. prompt-based fine-tuning has emerged as an effective alternative. this approach uses a prompt to guide the mlm in generating a specific token through masking a [mask] token in the prompt template, addressing the text classification tasks @xcite @xcite under lowresource conditions (i.e., few and zero-shot settings) through a verbalizer. as defined by @xcite , the verbalizer refers to the mapping from label words (e.g., "cryptanalysis") to the corresponding class (e.g., "cryptography"), serving as a projection function between the vocabulary and the class label space. however, in the context of classifying scientific literature, the complexity of scientific language and scarcity of fine-grained (i.e., a wide range of scientific fields that are labeled with sub-categories) or emerging topics make it hard to automatically classify crossdomain scholarly articles with limited training samples and manually created verbalizers @xcite . the goal of this paper is to address the challenge of multi-class classification in low-resource settings, specifically focusing on classifying scientific abstracts into different domains with only a limited number of labeled examples. we introduce a prompt-based fine-tuning approach enriched with domain knowledge as a new strategy for retrieving domain-adaptive label terms (i.e., scientific terms in various fields) without manual intervention. we enhance our approach for lowresource scenarios by retrieving scientific phrases from external knowledge bases (kbs) to expand label terms of the verbalizer @xcite from the token-level to term phrases. we fine-tune natural language inference (nli) models for semantic similarity search between retrieved label terms and class labels to select domain-related scientific phrases. our method differs from previous studies @xcite , which rely on word frequency filtering and are limited to singletoken verbalizer projection for text classification. given the complexity of scientific terminology (see appendix b for more details), we refine the traditional verbalization approach @xcite by integrating scientific terms through deploying a weight-aware label term mapping function. this approach improves the projection performance from mlm's predictions to probabilities of a specific class compared with prior studies @xcite @xcite . our approach consists of three stages: 1) retrieval of scientific terms, 2) label term filtering, and 3) prediction of scientific topics. initially, we use a cloze-style prompt and an input scientific abstract to guide the mlm to generate la-bel words to fill the [mask] token (figure 1 ). then, we use each class label as a query to retrieve class-related domain phrases (also denote as "label terms") from external kbs. to filter the potentially irrelevant terms gathered in the retrieval stage, we fine-tune both bi-encoder and cross-encoder models using the scinli dataset @xcite , enabling the selection of the most relevant domain phrases. finally, with the selected sets of knowledge-enriched scientific terms, we incorporate these label terms into the verbalizer to convert the mlm's prediction into a specific class through a semantic score-weighted average loss, enhancing the precision of the probability projections for the augmented verbalizer. our method extends beyond token-to-token verbalization by encompassing token-to-phrase verbalization that enriches the semantic meaning of scientific domain vocabulary. this broader scope allows for an advanced interpretation of scientific language and classifying emerging topics under weak supervision. in summary, our contributions are the presentation of: ## methodology our framework of sciprompt uses a two-stage approach for scientific text classification: (1) masked language modeling and (2) domain knowledge retrieval and filtering. ## experiments we present the experimental settings of sciprompt across scientific classification datasets in few and zero-shot scenarios. ## conclusion we introduced a knowledge-enhanced, promptbased fine-tuning framework for fine-grained scientific text classification using minimally or no labeled abstracts. acknowledging the complexity of domain knowledge within scientific literature, we employed a prompt-tuned mlm augmented with domain knowledge injection and semantic filtering. this approach enables the automatic extraction of domain-specific phrases and their integration into a weighted verbalizer for topic projection. our findings highlight the effectiveness of our methods over existing state-of-the-art models and standard full-set fine-tuning, particularly for emerging topic classification and scenarios requiring high levels of topic granularity. notably, sciprompt demonstrates competitive accuracy compared to the advanced llama 2 70b model in the zero-shot setting, showing its potential to categorize scholarly topics with a lightweight and efficient approach.