S peech GPT : Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023)

Abstract: Multi-modal large language models are regarded as a crucial step towards Artificial General Intelligence (AGI) and have garnered significant interest with the emergence of ChatGPT. However, current speech-language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. In this paper, we propose SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-modal content. With discrete speech representations, we construct SpeechInstruct, the first large-scale cross-modal speech instruction dataset. Additionally, we employ a three-stage training strategy that includes modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The experimental results demonstrate that SpeechGPT has an impressive capacity to follow cross-modal human instructions and highlight the potential of handling multiple modalities with one model. Code and models are available in https://github.com/0nutation/SpeechGPT . Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/ .

## introduction large language models @xcite have performed astonishingly on various natural language processing tasks. meanwhile, multi-modal large language models, such as gpt-4, palm-e @xcite , and llava @xcite , have explored the ability of llms to understand multi-modal information. however, a significant gap exists between current llms and general artificial intelligence (agi). first, most current llms can only perceive and understand multimodal content but cannot spontaneously generate multi-modal content. second, continuous signals ## speechinstruct construction due to the limitations in publicly available speech data and the lack of variety of speech-text tasks, we construct speechinstruct, a speech-text crossmodal instruction-following dataset. this dataset consists of two parts, the first part is called cross-modal instruction, and the second part is called chain-of-modality instruction. the construction process of speechinstruct is illustrated in figure 2 . ## conclusion this work presents speechgpt, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-modal content. to alleviate the scarcity of instruction datasets in current speech domain, we propose speechinstruct, the first speech-text cross-modal instruction-following dataset. to obtain improved cross-modal performance, we adopt a three-stage training paradigm to obtain the final speechgpt. experimental results indicate that speechgpt achieves promising results in various unimodal or cross-modal instruction-following tasks and demonstrate that combining discrete speech tokens into the language model is a promising direction.