Unlocking Speech Instruction Data Potential with Query Rewriting (2025)

Abstract: End-to-end Large Speech Language Models (**LSLMs**) demonstrate strong potential in response latency and speech comprehension capabilities, showcasing general intelligence across speech understanding tasks. However, the ability to follow speech instructions has not been fully realized due to the lack of datasets and heavily biased training tasks. Leveraging the rich ASR datasets, previous approaches have used Large Language Models (**LLMs**) to continue the linguistic information of speech to construct speech instruction datasets. Yet, due to the gap between LLM-generated results and real human responses, the continuation methods further amplify these shortcomings. Given the high costs of collecting and annotating speech instruction datasets by humans, using speech synthesis to construct large-scale speech instruction datasets has become a balanced and robust alternative. Although modern Text-To-Speech (**TTS**) models have achieved near-human-level synthesis quality, it is challenging to appropriately convert out-of-distribution text instruction to speech due to the limitations of the training data distribution in TTS models. To address this issue, we propose a query rewriting framework with multi-LLM knowledge fusion, employing multiple agents to annotate and validate the synthesized speech, making it possible to construct high-quality speech instruction datasets without relying on human annotation. Experiments show that this method can transform text instructions into distributions more suitable for TTS models for speech synthesis through zero-shot rewriting, increasing data usability from 72% to 93%. It also demonstrates unique advantages in rewriting tasks that require complex knowledge and context-related abilities.

## introduction llms have demonstrated powerful performance in general intelligence, profoundly changing the † corresponding author. benefiting from abundant automatic speech recognition (asr) datasets @xcite @xcite , early work aligned speech and text modalities by having models repeat or recognize linguistic information in speech through textual instructions @xcite . to enhance the model's understanding of paralinguistic information in speech, traditional tasks in the speech domain (such as speaker classification, speech entity recognition, etc.) were integrated into training @xcite @xcite . these training methods, which treat the speech modality as files rather than instructions, cause the model to lack the ability to follow speech instructions and lead to hallucinations due to the severe bias in task distribution. to unlock the model's potential to follow speech instructions, previous approaches constructed speech instruction datasets by continuing the linguistic information of speech through llms to restore the model's instructionfollowing capabilities @xcite . however, due to the gap between the generated results of llms and human-annotated ground truth answers, using such continuation methods may further amplify these shortcomings. unlike the image domain, which has abundant human-annotated data @xcite , constructing large-scale, manually narrated, and annotated speech instruction datasets from scratch is challenging due to the high costs of collection and annotation. as a result, using speech synthesis to construct datasets becomes a robust choice after weighing the pros and cons. however, due to differences between tts models and human speech narration, as well as hallucination issues during the speech synthesis process, it is necessary to verify whether the synthesized speech is linguistically equivalent to the text. on the other hand, since tts models have a limited vocabulary, they cannot accurately convert out-of-distribution text, such as compound words, abbreviations, or mathematical formulas, into speech, leading to the loss of linguistic information. previous work has shown that the semantic similarity between two texts can be calculated using embedding models, demonstrating better robustness compared to methods like wer and more closely aligned with how humans assess textual similarity @xcite . some high-quality human-annotated datasets achieve superior annotation results by integrating the opinions of multiple annotators, making the sample distribution more representative of real-world scenarios @xcite . meanwhile, query rewriting optimizes the text distribution without significantly altering the semantics, making it better aligned with a specific distribution @xcite . inspired by this, we propose a query rewriting method with multi-llm knowledge fusion, along with multi-agent annotation and data quality validation. specifically, we leverage llms' generalization ability in zero-shot tasks to guide the rewriting of text instructions to fit the training distribution of tts models. additionally, we use multiple distinct llms to rewrite the text from different perspectives. next, we automatically extract linguistic information from the speech using multiple models and calculate its average similarity to the original text in the embedding space to avoid annotation errors and achieve semantically optimal results. finally, we fuse the knowledge of different llms in this zero-shot rewriting task to tackle challenging rewrites that require complex knowledge. experimental results show that our method increased data usability from 71% to 93% and improved the semantic similarity between the linguistic information in speech and the original text by 5%. the main contribution of this paper can be summarised as follows: speech-text alignment training in lslms aligning speech and text modalities is crucial for building the speech understanding capability of end-to-end lslms. benefiting from abundant asr datasets @xcite @xcite , early work aligned the two modalities by instructing the model to repeat or recognize the linguistic information in the speech @xcite @xcite . however, due to the severely biased task distribution, the model ignored following the instructions in the speech and defaulted to performing language recognition @xcite . recent work has employed a continuation approach to construct speech instruction data, aiming to restore the model's ability to follow spoken instructions @xcite . due to the gap between the language model's generated outputs and real human responses, using the continuation method to train the model can further amplify this deficiency @xcite . in this work, we propose a high-quality speech instruction synthesis method to address this issue. ## problem formulation our goal is to automatically obtain synthesized speech that is linguistically equivalent to the text. for a given tts model and text instruction c o , we can get the synthesized speech s o . ideally, we can get the linguistic information co in s o by asr model, it should satisfies as shown in figure 1 , we present some recognition errors in asr models. due to the inherent gap between asr models and humans in speech recognition, it is challenging to get co that satisfies equation (1), even for speech that contains equivalent linguistic information. therefore, using the strict linguistic equivalence condition in equation (1) to evaluate the quality of synthesized speech can lead to inappropriate data rejection and cause a shift in the dataset distribution. inspired by the work on semantic textual similarity @xcite , we use the similarity between co and c o as a criterion for judging their linguistic equivalence. given the similarity calculation method f , our goal is to get s to maximize the number of @xmath0, c). ## conclusion in this paper, we propose a query rewriting method based on multi-llm knowledge fusion, with multiagent annotation and validation for data quality. experiments demonstrate that this method consistently performs well across multiple datasets, improving the average data usability from 72% to 93%. through ablation studies, we analyzed the effectiveness of each component, and the results show that different llms exhibit a certain degree of orthogonality in this zero-shot task. moreover, using multiple annotation agents helps to better reduce data quality evaluation errors and improper data filtering caused by recognition and annotation errors of a single model. our work enables the automated construction of high-quality language instruction datasets.