A udio C hat L lama: Towards General-Purpose Speech Abilities for LLM s (2024)

Abstract: In this work, we extend the instruction-tuned Llama-2 model with end-to-end general-purpose speech processing and reasoning abilities while maintaining the wide range of original LLM capabilities, without using any carefully curated paired data. The resulting end-to-end model, named AudioChatLlama , can utilize audio prompts as a replacement for text and sustain a conversation. Such a model also has extended cross-modal capabilities such as being able to perform spoken question answering (QA), speech translation, and audio summarization amongst many other closed and open-domain tasks. This is unlike prior approaches in speech, in which LLMs are extended to handle audio for a limited number of pre-designated tasks. On both synthesized and recorded speech QA test sets, evaluations show that our end-to-end approach is on par with or outperforms cascaded systems (speech recognizer + LLM) in terms of modelling the response to a prompt. Furthermore, unlike cascades, our approach can interchange text and audio modalities and intrinsically utilize prior context in a conversation to provide better results.

## introduction large language models @xcite @xcite ) (llms) have, due to their flexibility and extended capabilities, proven themselves highly performant on a wide range of natural language tasks, including open-domain tasks which might require world-knowledge. such generative tasks include text summarization, text and code generation, information retrieval, and machine translation among others @xcite . despite the impressive performance of these models, there is a mismatch between the criteria they were trained on and * work done during internship at meta ai. users' preferences @xcite . vanilla llms when prompted would often produce non-factual, unhelpful, or toxic materialthe model is not aligned with user preferences @xcite @xcite . this has led to a wide range of work on aligning the behavior of large language models with the intent of users @xcite . on a different front, there has been a notable surge in work extending the end-to-end capabilities of large language models (llms) to other modalities such as audio, with the aim of enabling them to process information that was previously difficult to encapsulate purely with text @xcite @xcite @xcite . for example, audio has the capacity to encapsulate a diverse array of emotions within a person's speech, while images have the ability to depict structures and placement of objects, which could be significantly more complex to convey through text. the work of @xcite aligned a large pre-trained visual transformer @xcite to the palm llm @xcite using a dedicated dataset for robotics with natural language. similarly, @xcite aligned a pre-trained visual model to the large language model vicuna, a fine-tuned version of llama @xcite , using a carefully curated paired image-text dataset to enable reasoning. in the audio domain, @xcite proposed ltu, an extension of llama with an aligned audio encoder trained on a curated audio question-answering corpus. this enabled ltu to reason with and understand sounds but still lacked speech recognition abilities. furthermore, @xcite used a similar data processing approach to curate a dataset, enabling the resulting system (with dual audio encoders) to perform a wide range of tasks including speech recognition. ## audiochatllama: llm with general-purpose speech abilities there has been a significant amount of recent work on equipping llms with multi-modal processing capabilities. the current standard approach revolves around two components: (1) a pre-trained encoder for the new modality and (2) paired data which is used in aligning the encoder with the language model for the particular tasks the joint system should solve @xcite @xcite @xcite @xcite . furthermore, most of these works use vanilla llms, instead of instruction-tuned versions which are often more aligned with user preferences in terms of usability and content generation. our goal is to create a system that can be prompted with audio as a direct replacement for text and allow for a user to use speech to converse while maintaining the llm original capabilities. furthermore, we aim to achieve this without curating dedicated paired datasets for extending these capabilities to the speech domain. to achieve this we first opt to equip an instruction-tuned llm with an audio encoder to enable it to directly process speech representations. second, instead of relying on utilizing paired datasets, we make use of a modal-invariance trick: whether the input to the llm is a string of text or an audio recording, the response to both prompts should be identical if the semantic information in the modalities is the same. the resulting system is named audiochatllama due to the extension of llama-2-chat to the speech conversational domain. the following subsections cover the architecture for our approach, the audio encoder for feature extraction, the type of large language model, and finally, how we align the audio encoder to the llm while achieving all the outlined goals. ## results this section reports empirical results of evaluating the ability of audiochatllama in various settings. first, we measure the perplexity of the speechprompted llm on the responses generated by the text-prompted llm. comparing to a cascade of asr + llm may provide some insight on how well the end-to-end approach performs. second, starting from a text-based qa dataset, we synthesize spoken questions using a text-to-speech system. we then conduct human evaluation on responses from the cascaded baseline and audiochatllama. evaluators judge the response quality by comparing with reference answers. to further understand the performance of the proposed model on real speech, we also perform human evaluation on a real spoken qa dataset. finally, since our proposal can interact with speech it has many more capabilities which are showcased in the last subsection. ## limitation the modal-invariance approach used in our work to extend text capabilities to the speech domain has worked effectively. however, the resulting model still has limited audio understanding. a possible next step would involve extending these capabilities to generic audio understanding and reasoning in addition to the capabilities presented in this paper. furthermore, the first step in achieving this would be to replace the small conformer audio encoder with one of the many available more robust self-supervised trained audio encoders such as wav2vec2 @xcite or hubert @xcite . that a small conformer trained on a limited amount of data still works is a good stepping stone for stronger audio models that can encode audio into richer representations useful for a wider range of tasks. ## conclusion in this work, we extend an instruction-tuned large language model with end-to-end speech processing and reasoning abilities while maintaining its wide range of capabilities, without using any carefully curated paired data. the result, referred to as audiochatllama, is a large language model that can utilize audio prompt as a replacement of text and sustain a conversation in an end-to-end manner. such a model also has extended crossmodal capabilities such as being able to perform spoken qa, speech translation, and audio summarization. finally, unlike alternative approaches that rely on standard speech recognition systems which struggle to recognize certain rare words, the prior context in a conversation can guide our system in various recognition and reasoning tasks. deyao zhu, jun chen, xiaoqian shen, xiang li, and mohamed elhoseiny. 2023. minigpt-4: enhancing vision-language understanding with advanced large language models. arxiv preprint arxiv:2304.10592.