Self-Powered LLM Modality Expansion for Large Speech-Text Models (2024)

Abstract: Large language models (LLMs) exhibit remarkable performance across diverse tasks, indicating their potential for expansion into large speech-text models (LSMs) by integrating speech capabilities. Although unified speech-text pre-training and multimodal data instruction-tuning offer considerable benefits, these methods generally entail significant resource demands and tend to overfit specific tasks.This study aims to refine the use of speech datasets for LSM training by addressing the limitations of vanilla instruction tuning. We explore the instruction-following dynamics within LSMs, identifying a critical issue termed speech anchor bias—a tendency for LSMs to over-rely on speech inputs, mistakenly interpreting the entire speech modality as directives, thereby neglecting textual instructions.To counteract this bias, we introduce a self-powered LSM that leverages augmented automatic speech recognition data generated by the model itself for more effective instruction tuning. Our experiments across a range of speech-based tasks demonstrate that self-powered LSM mitigates speech anchor bias and improves the fusion of speech and text modalities in LSMs. Data, code and scripts are freely available at https://github.com/ytf-philp/Self-powered-LSM.

## introduction recent advances in large language models (llms) @xcite @xcite have achieved remarkable performance across numerous tasks (openai, 2023). to further extend the capabilities of llms, many efforts @xcite @xcite have been made to integrate llms with multi-modal encoders, aiming to endow models with multimodal capabilities. speech, as a typical * corresponding author modality, shares similar semantics with text and has become a widely studied area. to expand the speech capabilities of llms, several studies adopt multitask learning @xcite or unified speech-text pre-training @xcite . while yielding notable performance, these methods require extensive computational resources. to address this, @xcite construct multimodal speech instruction datasets to fine-tune llms for developing large speech-text models (lsms). however, vanilla datasets often reveal lsms' deficiencies in following instructions @xcite . to avoid task-specific overfitting and enhance generalizability, careful development of datasets and methodologies is necessary. given the prevalence of automatic speech recognition (asr) datasets, using this data to build multimodal speech instruction datasets is straightforward. thus, we question whether lsms can be fully developed solely by collecting this type of training data. to achieve this, this study first explores the attention dynamics between speech inputs and instructions, identifying inherent training challenges presented by vanilla asr training. our analysis reveals a pronounced tendency for lsms trained with speech instructional data to overly concentrate on speech, neglecting instructions. this predisposition causes lsms to mistakenly interpret the entire speech modality as instruction, limiting the model's responses to scenarios that closely mirror its training data. we define this phenomenon as speech anchor bias. motivated by this finding, we argue that emphasizing the role of instructions during the training process is crucial for enhancing the instructionfollowing ability of lsms. to achieve this, we introduce a novel self-powered lsm that utilizes augmented instructional data generated by the model itself to facilitate the expansion of modality capabilities. experiments conducted on asr, speech translation (st), speech-language understanding (slu), and question answering (qa) demonstrate that our method, even when applied solely to asr datasets, effectively improves lsm performance across diverse tasks, thereby alleviating speech anchor bias and enhancing the integration of speech and text modalities in lsms the main contributions of this paper are: ## conclusion in this study, we aim to expand the capabilities of llms in speech modalities by exclusively utilizing asr data. to achieve this, we explore the instruction-following dynamics within lsms, identifying a significant issue that we term speech anchor bias. this bias highlights lsms' tendency to unduly emphasize speech inputs. to rectify this, we developed a novel self-powered method that leverages augmented asr data generated by the models to enhance lsms' instruction-following capabilities. our experiments across multiple speech tasks demonstrate that the self-powered lsm significantly mitigates the speech anchor bias, substantially improving lsms' fidelity to instructions.