PSLM : Parallel Generation of Text and Speech with LLM s for Low-Latency Spoken Dialogue Systems (2024)

Abstract: Multimodal language models that process both text and speech have a potential for applications in spoken dialogue systems. However, current models face two major challenges in response generation latency: (1) generating a spoken response requires the prior generation of a written response, and (2) speech sequences are significantly longer than text sequences. This study addresses these issues by extending the input and output sequences of the language model to support the parallel generation of text and speech. Our experiments on spoken question answering tasks demonstrate that our approach improves latency while maintaining the quality of response content. Additionally, we show that latency can be further reduced by generating speech in multiple sequences. Demo samples are available at https://rinnakk.github.io/research/publications/PSLM.

## introduction spoken dialogue systems have been developed for many years to achieve natural human-computer interaction @xcite @xcite . traditionally, these systems consist of several components: automatic speech recognition (asr), response generation (rg), and text-to-speech (tts). various methods for rg have been proposed with the advancements in large language models (llms) @xcite . more recently, the application of llms to asr (e.g., @xcite @xcite ) and tts @xcite has attracted much attention, leading to the development of multimodal llms capable of end-to-end spoken language communication @xcite . @xcite proposed speechgpt, an llm that receives speech questions (sq) as speech tokens, which are discrete representations extracted from raw waveforms, and sequentially generates text questions (tq), text answers (ta), and speech answers (sa). figure 1 (a) illustrates their approach called chain-of-modality (com) prompting. spectron @xcite follows this prompting style but directly handles speech spectrograms. although these methods can generate high-quality responses, they face two major challenges in terms of response latency. first, generating sa requires the prior generation of tq and ta. second, speech sequences are much longer than text sequences 1 . in this study, we propose parallel speech language model (pslm), an llm with multiple inputoutput sequences to handle both text and speech tokens, enabling their parallel generation. to emphasize their parallel processing capabilities, we will refer to these sequences as "streams". as described in figure 1 (b), pslm begins to generate sa immediately after the end of sq tokens, which can reduce overall latency. this leads to our first research question (rq1): can pslm improve latency while maintaining the response quality achieved by com prompting? additionally, we address the second challenge by introducing multiple speech streams to decode multiple speech tokens in a single step, as described in figure 1 @xcite . this brings us to the second research question (rq2): do multiple speech streams sacrifice the response quality? addressing these questions will pave the way for more advanced and responsive applications of spoken dialogue systems. ## conclusion in this study, we proposed the parallel speech language model (pslm), an llm capable of generating text and speech tokens in parallel with multiple input-output streams, and investigated its impact on response quality and overall latency. the experimental evaluations on spoken question answering demonstrated that the proposed method significantly reduces latency compared to existing methods while maintaining response quality. future work includes verifying the effectiveness of the proposed method on larger datasets and real speech data. additionally, extending the proposed method to multi-turn dialogues is an important research direction.