Making Pre-trained Language Models Better Learn Few-Shot Spoken Language Understanding in More Practical Scenarios (2023)

Abstract: Most previous few-shot Spoken Language Understanding (SLU) models typically need to be trained on a set of data-rich source domains and adapt to the target domain with a few examples. In this paper, we explore a more practical scenario for few-shot SLU, in which we only assume access to a pre-trained language model and a few labeled examples without any other source domain data. We concentrate on understanding how far the few-shot SLU could be pushed in this setting. To this end, we develop a prompt-based intent detection model in few-shot settings, which leverages the BERT original pre-training next sentence prediction task and the prompt template to detect the user’s intent. For slot filling, we propose an approach of reconstructing slot labels, which reduces the training complexity by reducing the number of slot labels in few-shot settings. To evaluate the few-shot SLU for a more practical scenario, we present two benchmarks, FewShotATIS and FewShotSNIPS. And a dynamic sampling strategy is designed to construct the two datasets according to the learning difficulty of each intent and slot. Experiments on FewShotATIS and FewShotSNIPS demonstrate that our proposed model achieves state-of-the-art performance.

## introduction spoken language understanding (slu) is one of the fundamental modules for task-oriented dialogue systems, which mainly includes two sub-tasks, intent detection and slot filling. the remarkable success of most neural slu models typically relies on a large quantity of training data @xcite @xcite . however, acquiring large amounts of annotated data for domain-specific is arduous and expensive in practical applications. situations like few or even no training data may happen in a brand-new application, which motivates us to address the challenge of the slu module in few-shot settings. the previous few-shot slu studies mainly focused on semi-supervised learning @xcite @xcite and metric learning @xcite @xcite @xcite @xcite @xcite . the models need to be trained on source domains with abundant data and then adapt to the data-scarce target domain. straying from the pattern by making minimal assumptions about available resources, we explore a more practical scenario for few-shot slu, in which we only use moderately sized pre-trained models, such as bert @xcite or roberta @xcite , and only a tiny amount of annotated examples to fine-tune the pre-trained model. the settings are pretty attractive as (1) the few-shot settings conform more to real application scenarios, as it is straightforward to get a few annotated data (e.g., 10 examples); (2) such models are not demanding hardware resources for training; and (3) the development of prompt tuning has brought a novel paradigm for few-shot learning, and updating parameters of the pre-trained model typically leads to better performance @xcite @xcite . we focus on exploring what the performance of the few-shot slu model can achieve without any other source domain data. we revisit existing few-shot slu benchmarks @xcite @xcite and find that the settings of them tend to deviate more from practical application scenarios. the main reasons are as follows. (1) these benchmarks provide data-rich source domains for training, such as snips @xcite and clinc @xcite . however, in practical applications, there might be little or even no training data available. (2) some benchmarks typically uniformly random sample k examples for each intent and slot. however, in practical applications, the learning complexity varies with different intents and slots and may necessitate different amounts of training data. to fill these gaps, we construct two multidomain few-shot slu benchmarks, fewshotatis and fewshotsnips, without any source domain data, which are based on atis @xcite and snips @xcite , respectively. in addition, we design a dynamic sampling strategy that allocates the number of samples depending on the evaluation metrics of each intent and slot. the sampling strategy is more reasonable and adheres to practical scenarios since slight variations in the training data can have a major impact on the model's performance in few-shot settings. the proposed benchmarks provide a fair comparison between various methods on common ground and measuring progress in practical scenarios. recently, prompt tuning achieved competitive results with only a limited amounts of training data, such as prompt tuning based on bert with the masked language model (as shown in figure 1(a) ), which has been proven to be effective for text classification @xcite @xcite . however, it suffers from the following challenges when employed for intent detection. moreover, there are some challenges for the slot filling task in few-shot settings. (1) the bio labeling format @xcite is typically utilized in standard datasets for slot filling, as shown in figure 2 (a). in the case of sufficient data, the beginning labeling 'b-' of the entity provides supervised information to the model to detect the entity boundaries. nevertheless, in few-shot settings, the excessive amount of slot labels in the labeling format exacerbates the difficulty of the model training. to address the aforementioned challenges, we proposed a bert-nsp-prompt model for intent detection and a reconstructing slot labels approach for slot filling. in particular, bert-nsp-prompt leverages the next sentence prediction (nsp) task of bert and a constructed prompt template, to complete intent detection in the few-shot or even zero-shot settings, as shown in figure 1 @xcite . it is used to assess which intent description text is the most "fluent" following sentence of the user's dialogue. moreover, an approach of reconstructing slot labels is proposed to reduce the model training complexity by reducing the number of slot labels in few-shot settings, as shown in figure 2(b) . we convert the bio slot labeling format to slot entity labeling format, which resulted in the slot category cutting in half. furthermore, we introduce the focal loss function @xcite to distinguish between similar slot labels. to sum up, our key contributions are as follows. ## related work most works use semi-supervised learning or metric learning to implement few-shot slu tasks @xcite @xcite @xcite . however, these models still require a set of datarich source domains. recently, prompt tuning has taken advantage of the powerful generalization ability of pre-trained language models and dramatically reduces the reliance on supervised data for downstream tasks @xcite @xcite . the paradigm narrows the gap between the pre-trained model and downstream tasks, which provides a novel insight into few-shot learning. we propose a simple and effective method based on pre-trained models to accomplish few-shot slu tasks. the proposed model does not have a transfer from the source domain to the target domain, nor does it use any additional data resources. it will serve as an essential baseline for future exploration. in addition, we believe the constructed fewshotatis and fewshotsnips datasets can comprehensively evaluate the models and inspire the research of the few-shot slu. ## datasets: fewshotatis and fewshotsnips in few-shot slu, existing benchmarks usually uniformly random sample k examples for each intent and slot, such as snips @xcite , and clinc @xcite . however, a reasonable and conform to practical scenarios sampling strategy is: suppose the total number of samples is limited; for easy-to-classify intents or slots, it will sample less than k samples; in contrast, it will sample more than k samples for hard-to-classify intents and slots. thus, we design a dynamic sampling strategy to simulate the "sampling-iterative" process, which assigns the number of samples according to the evaluation metrics of each intent and slot. since small changes in the training data may significantly affect the model's performance in few-shot settings, the advantages of the strategy will be magnified further. finally, we adopt the dynamic sampling strategy to build two new multidomain few-shot slu benchmarks, fewshotatis and fewshotsnips foot_0 , respectively. ## experiments as stated in the paper, the few-shot slu setting assumes access to a moderately sized pre-trained language model and a few labeled data, without the data from any other source domain. as such, moderately sized pre-trained models, namely bert and roberta, are employed as the baseline models. the details of implementation and evaluation metrics can be found in appendix c. the main reasons for not considering metric learning models as baseline models are as follows. (1) these models necessitate training on source domains with a voluminous dataset to adapt to the data-scarce target domain, which is not feasible in our few-shot slu setting. (2) the two presented benchmarks are with limited annotation and without source domain data, making it difficult for the previous metric learning models to be trained efficiently. as such, it is impossible to fairly compare the performances of the proposed and previous models on proposed two datasets. to further compare with the previous few-shot slu models, we attempt to conduct comparative experiments following the dataset setting of @xcite . the results are presented in appendix d. ## conclusion in this paper, we explore a more realistic scenario for the few-shot slu. there are two main contributions toward developing a few-shot slu. (1) we built two benchmarks, fewshotatis and few-shotsnips, to simulate the few-shot slu task in a more realistic scenario. (2) we develop bert-nsp-prompt, which utilizes bert's nsp task with a prompt template to detect user intent. in addition, we proposed a reconstructing slot label approach to reduce the difficulty of training the classifier by reducing the number of labels. experimental results indicate that the proposed model achieves the sota performance and endows the slu module with solid generalization ability. there are two interesting directions for future work in the few-shot setting. (1) fully explore the close relationship between intent and slot to improve the performance of slu. (2) it is an idea worth exploring that reduces the complexity of training models.