Speech-based Slot Filling using Large Language Models (2024)

Abstract: Recently, advancements in large language models (LLMs) have shown an unprecedented ability across various language tasks. This paper investigates the potential application of LLMs to slot filling with noisy ASR transcriptions, via both in-context learning and task-specific fine-tuning. Dedicated prompt designs and noise-robust LoRA fine-tuning are proposed to improve the robustness of LLMs for slot filling with noisy ASR transcriptions. Moreover, a linearised knowledge injection (LKI) scheme is also proposed to integrate dynamic external knowledge into LLMs. Experiments were performed on SLURP to quantify the performance of LLMs, including GPT-3.5-turbo, GPT-4, LLaMA-13B, LLaMA-2-13B and Vicuna-13B (v1.1 and v1.5) with different ASR error rates. The use of the noise-robust fine-tuning together with LKI for Vicuna-13B-v1.5 achieved 6.7% and 17.6% absolute SLU-F1 improvements compared to a fully fine-tuned Flan-T5-XL model on the limited data setup and the zero-shot setup respectively.

## introduction slot filling, as an important sub-task of spoken language understanding (slu), is a crucial component in conversational ai such as spoken dialogue systems. it requires the extraction and understanding of pertinent information in the user's speech query. accurate extraction of slot values from the query speech is indispensable for accurate response generation and is challenging with limited annotated data and noisy asr transcriptions. in particular, domain-specific named entities that are crucial to accurate information extraction, usually have high error rates with a generic asr system. although this problem can be mitigated by training systems on data in the target domain, it can be expensive to construct dedicated large-scale training data for a specific slu task as it requires an extensive labelling effort and domain expertise @xcite @xcite . this data sparsity problem can be addressed by transfer learning with pre-trained language models (plms) @xcite , especially with large language models (llm). recent advancements in llms, such as gpt-4 @xcite and the llama series @xcite , have been shown to exhibit human-level reasoning ability for natural language tasks even without taskspecific fine-tuning of the model parameters, which is known as the emergence of llms. this is usually achieved by conditioning the model generation process on a prompt containing examples or a task description, referred to as in-context learning. although effective, studies have also shown that llms struggle with accurate fine-grained content extraction such as slot-filling @xcite @xcite , and tend to overly extrapolate beyond the examples and task descriptions in the prompt, especially when evaluated using text-based quantitative metrics. this necessitates the use of dynamic contextual knowledge to guide and confine the generation @xcite , as well as efficient task-specific fine-tuning with limited data @xcite . moreover, as slot-filling relies on the quality of asr transcriptions @xcite @xcite , it is essential to improve the performance of llms with noisy asr outputs. this paper aims to apply llms for slot filling with different asr error rates under limited data scenarios. a dedicated prompt design specifically targeting llms is proposed featuring task descriptions with examples, linearised dynamic external knowledge and multiple alternative asr hypotheses. in addition to the one-best hypothesis, the proposed prompt design incorporates multiple hypotheses in the form of an n -best list. moreover, by leveraging a pre-defined knowledge base (kb) @xcite , dynamic knowledge found in the n -best list is also linearised into text and used in the prompt to provide the necessary constraint to guide the language generation foot_0 . llms can be fine-tuned with low-rank adaptation (lora), and the use of alternative hypotheses and lki enables a noise-robust lora fine-tuning to further improve the model's robustness to asr errors. experiments were performed using slurp data, where particular attention was paid to the performance of in-context learning, supervised few-shot learning, and zero-shot learning for unseen slot types. several llms were investigated, including gpt-3.5-turbo and gpt-4 for in-context learning and llama and vicuna models as widely-used open-source models for fine-tuning. different sizes of whisper asr models were adopted as a group of generic off-the-shelf models to provide transcriptions with different asr error rates. experiments showed that vicuna-13b-v1.5 achieved an absolute 6.7% and 17.6% slu-f1 increase using the noise-robust lora fine-tuning together with lki compared to the fully fine-tuned flan-t5-xl on few-shot and zero-shot scenarios respectively, even though the number of trainable parameters of the latter was much larger. the main contributions of this paper can be summarised as follows. ## conclusions this paper investigated the performance of llms for slot filling with noisy speech transcriptions. it quantified the performance of six different llms using asr transcriptions from four different sizes of whisper asr models and proposed a dedicated prompt design, a noise-robust lora fine-tuning approach and a linearised knowledge integration (lki) scheme. the prompt design with few-shot examples enabled gpt-4 to outperform a gpt-2 model fine-tuned on 20 times more data. vicuna-13b-v1.5 achieved an absolute 6.7% and 17.6% slu-f1 increase using both the noise-robust lora fine-tuning and lki compared to the fully finetuned flan-t5-xl model on few-shot and zeroshot scenarios respectively, while the number of the main limitation of using lki and n -best hypotheses for training is the context length of llms. when sequence length becomes much longer by incorporating more examples or more hypotheses, the efficacy of long context is inherently limited by the effective spans of the attention mechanisms in llms. even with the vicuna-13b-v1.5 model which supports a sequence length of 4k tokens, a clear reduction in improvements was found when adding much more information into the context. therefore, future work needs to be done on improving the effectiveness of the context instead of only increasing the maximum allowed sequence length. due to resource constraints, experiments were only conducted on the slurp corpus. further studies will be carried out in the future on other speech-based slot-filling corpora using llms.