Synergistic Augmentation: Enhancing Cross-Domain Zero-Shot Slot Filling with Small Model-Assisted Large Language Models (2025)

Abstract: In real-world scenarios, cross-domain slot filling in spoken language understanding remains a significant challenge due to data scarcity. Previous works exhibit limited generalization ability in the target domain, demonstrating effective knowledge transfer only on seen slots while performing poorly on unseen slots. Although large language models (LLMs) can alleviate this issue to some extent, they underperform on seen slots compared to small models. To address these challenges, we introduce a novel framework that harnesses the power of a small model to augment the inferential capabilities of LLMs without additional training. Initially, we utilize target domain samples synthesized by LLMs as pre-calculated demonstrations, which are curated and chosen using confidence metrics derived from a small model. We further extract slot predictions from the small model to fully exploit its robust learning of familiar slots. Finally, during the inference process for test inputs, we integrate these demonstrations and slot prediction insights as references to enhance the slot filling performance of LLMs. Experiments on a slot filling dataset and a NER dataset including eight cross-domain settings show our framework achieves the best results. Our codes are publicly available at https://github.com/SIGSDSscau/SLSF.

## introduction slot filling is a core component in dialogue systems that aids in understanding user needs, directly impacting task completion rates and user experience. the task involves extracting and categorizing slot entities (such as artist or playlist) from user utterances. previous slot filling studies @xcite @xcite use a single-domain training and evaluation learning paradigm. thanks to the rapid advancements in deep learning and the abundant labeled figure 1 : the average performance of different slots across seven different domains on snips using four methods. "all", "seen", and "unseen" respectively represent all slots, slots that appear in both the source and target domains, and slots that only appear in the target domain. "zero data" means there are no examples to select, while "source data" indicates selecting examples from the source domain. traditional methods @xcite @xcite @xcite involve training a model in the source domain and achieving good performance in the target domain. however, they can only learn features of unseen slots from a limited number of similar seen slots, which leads to low unseen slot performance, as shown by adae in figure 1 . with the increasing capabilities of llms @xcite @xcite , we test their performance on different slots in cdsf under various settings, as shown in figure 1 . although llms perform exceptionally well on unseen slots, their performance on seen slots still falls short compared to small models. in this paper, we aim to explore how small models assist llms in integrating their own strengths and enhancing llms' overall performance without additional training. specifically, we do not use source domain samples as demonstrations. as shown in figure 1 , while the presence of slots in these samples similar to the target domain can enhance llms' performance, there are some irrelevant noisy slots or text that degrade performance on unseen slots. so we utilize samples generated by llms as demonstrations that are similar to the target domain. however, llms suffer from hallucination issues, leading to inevitable noisy synthetic samples. therefore, we proposed a data curation mechanism to effectively alleviate the harmful synthetic sample problem. our approach is inspired by the principles of learning theory that show the state changes of samples during training, offering crucial insights into the sample's value @xcite @xcite . a synthetic sample that is dynamically mispredicted or has ambiguous predictions could potentially harm a model. with a curation model trained on the source data, we analyze the learning evolution of synthetic samples and curate these by evaluating their evolution of confidence. based on this, we select suitable demonstrations from the curated dataset for each test input. to further leverage the robust learning of familiar slot types by the small model, we provide the small model's predictions to llms as references. finally, by combining the selected highquality synthetic demonstrations, small model's prediction and test input, we input them into llms to obtain more accurate final prediction results. overall, our main contributions can be summarized as follows: (1) to the best of our knowledge, we are the first to explore completing cdsf using llms' inference without training it. (2) we propose a method that leverages llms to synthesize samples as demonstrations of in-context learning, enhancing its understanding of slots in the target domain. (3) we propose a novel framework that leverages small models to augment the slot filling capability of llms. in our framework, we explore different strategies to curate synthesized samples, select samples with relatively high confidence as demonstrations for llms learning and provide slot prediction as reference from small models, leveraging the small model's effective learning of familiar slots. (4) experiments demonstrate that the performance of our proposed framework improves sig-nificantly on different slots and eight cross-domain settings in cdsf compared to baselines. ## related work cross domain slot filling. previous approaches @xcite @xcite utilize meta-information such as slot descriptions and slot instances to capture the semantic relationship between slot types and the user query. however, these models only learn surface mappings of slot types between different domains and perform poorly on unseen slots in the target domain. to alleviate this problem, recent approaches @xcite adopt a contrastive learning framework to learn a generalized feature representation. combining learnable prompts and slot descriptions @xcite and proposing a prompt-based hierarchical pipeline with three innovations @xcite achieves more accurate cross-domain performance. nevertheless, the generalization capability of these models is limited due to a lack of learning in the target domain slots. aimed at enhancing this ability, some methods @xcite treat slot filling tasks as machine reading comprehension tasks to enhance the semantic interaction between slots and user queries, and further pre-train models on large-scale external mrc datasets. other methods @xcite frame the task as a language generation task to leverage the rich pretraining knowledge of large-parameter generative models. both of these methods introduce general knowledge to better answer slot values corresponding to certain slots lacking knowledge. however, their performance on unseen slots still falls short. moreover, recent studies @xcite @xcite of llms show strong generalization in zero-shot and few-shot scenarios for different tasks. so we explore the performance of llms on our task, surpassing previous methods on unseen slots. however, llms still fall short in overall performance on cdsf @xcite . therefore, we leverage small models to supplement the performance deficiencies of llms. demonstrations in in-context learning. some works explore generating various demonstrations in icl @xcite . @xcite use llms to make predictions on the unlabeled ner corpus as demonstrations. however, cdsf has a more challenging zero-shot setting, query: is it nice now in beda slot filling: slot type |slot value timerange |now city |beda query: what is the weather going to be in seattle wa next month slot filling: slot type |slot value |proability timerange |next month |0.267 city |seattle |0.193 state |wa |0.258 small model as curation model d cur demonstrations description of slot filling task small model prediction step 4: inference with demonstrations and small model prediction will it be cloudy in tatamy timerange 0 0 0 0.17 (low) 0 0.245 (high) city step 1: demonstrations generation step 2: demonstrations selection step 3: slot filling prediction from small model slot type |slot value weather |cloudy city |tatamy stage 1: given {slot type set} and {target domain}, select some most relative slot combinations. stage 2: first, given a slot combination and a domain, you need to provide a corresponding slot value for each slot type. then, using these related slot values, generate a user query with a given {tatget domain}. demonstrations small model prediction d cur query: what is the weather going to be in seattle wa next month slot filling: slot type |slot value |confidence timerange |next month |0.267 city |seattle |0.193 state |wa |0.258 retrieval: nearest avg confidence 0.2625 rerank: avg confidence (pre-calculated) query: what is the weather going to be in seattle wa next month slot filling: slot type |slot value timerange |next month city |seattle state |wa llm as data generator query: is it nice now in beda slot filling: slot type |slot value timerange |now city |beda query: is it nice now in beda slot filling: slot type |slot value timerange |now city |beda slot_type |slot_value |confidence timerange |cloudy |low city |tatamy |high the low confidence indicate that uncertain about the slot value. test input will it be cloudy in tatamy test input will it be cloudy in tatamy identify potential slot types from query and fill in the corresponding slot values for each slot type. the slot types should come from the provided list, and the slot values should come from the query. original data small model as prediction model llm as inference model where there are no samples from the target domain. therefore, we first use llms to generate samples of target domains as a demonstration corpus. previous methods randomly select examples from training data @xcite , which may hinder the potential of llms. so retrieval-based method @xcite @xcite ) is intensively investigated. recent works @xcite @xcite adopt a select-then-rank framework, ranking the demonstrations selected by the k nearest neighbors (nearest, @xcite ) method based on different reordering techniques. @xcite use selfconsistency (sc, @xcite ) scores to rerank the selective samples. this approach involves using unlabeled corpora for multiple invocations of llms, consuming more resources and it is difficult to apply to our task where lack target domain data. furthermore, small models can be used to evaluate the quality of sample labeling @xcite . so we utilize small models to get label confidence of synthesized samples and use it to select suitable samples as demonstrations. ## method as shown in figure 2 , our framework is divided into four steps. in the first step, we use llms to generate target domain samples and use a small model to curate a high-quality synthetic dataset as a demonstration corpus. the corpus can be precalculated and saved for llms' inference. then, we retrieve some similar demonstrations using nearest method for each test input, then select more suitable demonstrations from these based on average confidence. next step, we obtain the slotfilling predictions for the test input from the small model to provide reference information for llms. in the final step, we combine suitable demonstrations, slot-filling predictions of a small model, and test input, and input them into llms to infer the final result. the last three steps are iterated to generate slot filling results for each test input. ## conclusion to fully leverage the complementary strengths of small models and llms, we propose a trainingfree, small model-assisted framework for crossdomain zero-shot slot filling with llms. our framework significantly improves performance across eight corss-domain settings on two datasets. comprehensive experimental analysis demonstrates the effectiveness and robustness of our framework.