FPT : Feature Prompt Tuning for Few-shot Readability Assessment (2024)

Abstract: Prompt-based methods have achieved promising results in most few-shot text classification tasks. However, for readability assessment tasks, traditional prompt methods lack crucial linguistic knowledge, which has already been proven to be essential.Moreover, previous studies on utilizing linguistic features have shown non-robust performance in few-shot settings and may even impair model performance.To address these issues, we propose a novel prompt-based tuning framework that incorporates rich linguistic knowledge, called Feature Prompt Tuning (FPT). Specifically, we extract linguistic features from the text and embed them into trainable soft prompts. Further, we devise a new loss function to calibrate the similarity ranking order between categories. Experimental results demonstrate that our proposed method FTPnot only exhibits a significant performance improvement over the prior best prompt-based tuning approaches, but also surpasses the previous leading methods that incorporate linguistic features. Also, our proposed model significantly outperforms the large language model gpt-3.5-turbo-16k in most cases. Our proposed method establishes a new architecture for prompt tuning that sheds light on how linguistic features can be easily adapted to linguistic-related tasks.

## introduction readability assessment (ra) is the task of evaluating the reading difficulty of a given piece of text @xcite . it has wide applications, such as choosing appropriate reading materials for language teaching @xcite , supporting readers with learning disabilities @xcite , and ranking search results by their reading levels @xcite . early works in ra mainly focused on designing handcrafted linguistic features such as word length (in characters/syllables), sentence length, and usage of different difficulty-level words. in recent years, ra has been dominated by neural network-based architectures @xcite . the key challenge of these methods is to learn a better text representation that can capture deep semantic features. current research has also explored different ways of combining linguistic features with pretrained language models (plms), achieving state-of-the-art results on numerous ra datasets @xcite . however, these studies have mainly focused on fine-tuning with a large amount of labelled data, while only a few studies have explored few-shot settings. prompt-based tuning, shown to be a powerful method for the classification task in the few-shot setting, makes full use of the information in plms by reformulating classification tasks as cloze questions. different prompt-based tuning strategies are illustrated in figure 1 . the hard prompt tuning applies a template with @xcite token to the original input and maps the predicted label word to the corresponding class @xcite . the performance is sensitive to the quality of template, which introduces time-consuming and labor-intensive prompt design and optimization. to address this problem, researchers propose soft prompt strategies, where continuous embeddings of trainable tokens replace the hard template and are optimized by training @xcite . despite the success in a range of text classification tasks, existing prompt-based tuning methods still suffer from inferior performance in ra. this might be attributed to the lack of linguistic knowledge which has been demonstrated to play a crucial role in ra @xcite @xcite . meanwhile, ra differs from general classification tasks in that there exists a notion of ranking order between classes. our intuition behind the utilization of linguistic knowledge is that the learned representations of different levels should preserve the similarity relationship analogous to that of original linguistic features of different levels. motivated by the above insights, in this paper, we propose a novel prompt-based tuning method that incorporates rich linguistic knowledge, called feature prompt tuning (fpt), as shown in the bottom of figure 1 . specifically, our methodology begins with extracting linguistic features from the text. these extracted features are subsequently embedded into feature prompts, functioning as trainable soft prompts. contrary to the conventional prompt tuning frameworks, our model can explicitly benefit from linguistic knowledge. furthermore, we devise a new loss function to calibrate the similarity relationships between the embedded features across different categories. our approach is straightforward and effective, offering wide applicability to other tasks where the importance of handcrafted features is emphasized. to verify the effectiveness of our proposed methods, we conduct extensive experiments on three ra datasets, including one chinese data @xcite and two english datasets, weebit @xcite and cambridge @xcite . by incorporating linguistic knowledge, our proposed model fpt improves significantly over other prompt-based methods. for instance, in the 2-shot setting, fpt brings a relative performance gain of 43.9% over the traditional soft prompt method on the chinese dataset and 5.50% on en-glish weebit. moreover, compared to other feature fusion methods, fpt outperforms the previous best method projecting feature (pf) @xcite ) by 43.19% on chinese data and 11.55% on english weebit data. also, we experiment on the large language model (llm), demonstrating the superiority of our approach on ra. we will make our code public available foot_0 . we summarize our contributions as follows: ## background we model ra as a text classification task. formally, a ra dataset can be denoted as @xmath0, y}, where x is the text set and y is the class set. each instance @xmath1, @xmath2, w 2 , ..., w |x| }, and is annotated with a label @xmath3, indicating the reading difficulty. ## conclusion inspired by the solid performance of prompt tuning on classification tasks and the importance of linguistic features in the ra task, we empirically investigated the effectiveness of incorporating linguistic features into prompt tuning for ra. we convert linguistic features of the input into soft tokens and utilize the similarity calibration loss to preserve the similarity relationship between classes before and after the transformation. the results show noticeable improvements over previous fusion methods and prompt-based approaches in the few-shot learning setting. the ablation study further illustrated that the proposed model benefits from linguistic features and additional similarity calibration. our proposed method, fpt, has demonstrated a new possibility of prompt tuning in an era dominated by llms, showcasing its undeniable significance and value in linguistic-related tasks.