PR o D eliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding (2024)

Abstract: Spoken Language Understanding (SLU) is a critical component of voice assistants; it consists of converting speech to semantic parses for task execution. Previous works have explored end-to-end models to improve the quality and robustness of SLU models with Deliberation, however these models have remained autoregressive, resulting in higher latencies. In this work we introduce PRoDeliberation, a novel method leveraging a Connectionist Temporal Classification-based decoding strategy as well as a denoising objective to train robust non-autoregressive deliberation models. We show that PRoDeliberation achieves the latency reduction of parallel decoding (2-10x improvement over autoregressive models) while retaining the ability to correct Automatic Speech Recognition (ASR) mistranscriptions of autoregressive deliberation systems. We further show that the design of the denoising training allows PRoDeliberation to overcome the limitations of small ASR devices, and we provide analysis on the necessity of each component of the system.

## introduction traditionally, spoken language understanding (slu) systems are pipelined systems of 1) an automatic speech recognition (asr) model, which transcribes audio to text, and 2) a semantic parsing model to convert text into structured frames. this results in some inherent limitations, such as cascading errors from asr mistranscriptions, larger overall systems size from having two models with redundant capabilities, and the inability to use audio cues for semantic parse generation. end-to-end (e2e) models hold promise of overcoming these limitations by removing the barrier between these two models and allowing audio information to be used in the semantic parse generation. in particular, deliberation-based e2e slu models @xcite improve explainability (transcript is produced by first-pass asr), support of complex compositional structure (via the general decoder), and quality (allows rewriting the asr transcriptions based on audio information). however, one downside of the existing deliberation-based e2e slu models is that they use an autoregressive (ar) decoder for generation, which yields high latency, especially for longer outputs. this precludes them from being used in the on-device setting, which requires low latency. previous semantic parsing models can yield very low latency with non-autoregressive (nar) decoders @xcite @xcite , however, these systems use extractive decoders and are therefore unable to correct mistranscriptions from asr systems. our study targets improving deliberation based slu systems to adapt them for on-device streaming use cases. in this work we present prodeliberation (parallel robust deliberation), a novel connectionist temporal classification (ctc) decoderbased e2e slu model. ctc decoders are commonly used for asr @xcite and have successfully been adapted for machine translation @xcite , but to the best of our knowledge, have never been used for semantic parsing. to further improve the rewrite capabilities of this model we introduce a denoising training regimen inspired by masked language modeling @xcite in which we corrupt the transcription text, requiring the model to correct the corruptions by relying on audio. the denoising training applies beyond slu and can be used with any downstream task which consumes asr transcriptions. we study prodeliberation and find that prodeliberation retains the quality of autoregressive parsers while achieving a 2-10x latency reduction across different asr model sizes. we compare prodeliberation to mask predict-based approaches @xcite and confirm that prodeliberation overcomes the quality shortcomings of mask predict. finally, we show that the denoising objective further boosts asr robustness by approximately 0.3%, thereby exceeding the quality of autoregressive models. we provide a summary of our approach in figure 1 ; concretely, our contributions are: ## prodeliberation in this work we extend deliberation with two critical components: first we introduce ctc decoding, which utilizes parallel decoding to optimize latency. however, we notice an increased error rate due to parallel decoding. in order to combat this we add the second key component: denoising, which improve model robustness during training. figure 2 shows the architecture of prodeliberation in comparison to the architecture of autoregressive deliberation. 2020 ) as a two-pass asr system to improve transcription quality. we describe these two passes in detail in the following. ## conclusion we presented a novel slu model, prodeliberation, which uses a ctc decoder and noise augmentation to train a parallelized and robust e2e slu model. this model achieves much lower latency than the state-of-the-art models while maintaining the ability to correct errors in the asr transcript. further, we analyzed the system's performance across two asr model sizes and provided ablations guiding the decisions made when designing prodeliberation. we also show that the denoising training for prodeliberation allows it to overcome the limitations of very small asr models. this represents a significant step forward in building e2e slu systems that are small enough and have low enough latency to run on edge devices.