OWSM - CTC : An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (2024)

Abstract: There has been an increasing interest in large speech models that can perform multiple tasks in a single model. Such models usually adopt an encoder-decoder or decoder-only architecture due to their popularity and good performance in many domains. However, autoregressive models can be slower during inference compared to non-autoregressive models and also have potential risks of hallucination. Though prior studies observed promising results of non-autoregressive models for certain tasks at small scales, it remains unclear if they can be scaled to speech-to-text generation in diverse languages and tasks. Inspired by the Open Whisper-style Speech Model (OWSM) project, we propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC). It is trained on 180k hours of public audio data for multilingual automatic speech recognition (ASR), speech translation (ST), and language identification (LID). Compared to encoder-decoder OWSM, our OWSM-CTC achieves competitive results on ASR and up to 24% relative improvement on ST, while it is more robust and 3 to 4 times faster for inference. OWSM-CTC also improves the long-form ASR result with 20x speed-up.We will publicly release our code, pre-trained model, and training logs to promote open science in speech foundation models.

## introduction the great success of large language models (llms) @xcite @xcite has sparked a growing interest in developing foundation models in various modalities. recent studies have explored different approaches towards multilingual and multi-tasking speech foundation models @xcite @xcite @xcite 2023e). openai whisper @xcite is a series of transformer encoder-decoder models trained on 680k hours of proprietary labeled audio. whisper achieves strong results in multilingual automatic speech recognition (asr), any-to-english speech translation (st), and spoken language identification @xcite . although it shows the effectiveness of large-scale (weakly) supervised pre-training, the full development pipeline, including training data details, is not publicly accessible. recent works have developed open whisperstyle speech models (owsm) @xcite with the aim of reproducing whisper-style training using public data and open-source toolkits. however, whisper and owsm adopt the encoderdecoder architecture, which generates text tokens given speech in an autoregressive manner. they might hallucinate during inference, and the speed can be slow. other models with decoder-only architectures, like audiopalm @xcite and viola @xcite , could suffer from the same issues due to autoregressive decoding. another type of work like google usm @xcite and meta mms @xcite uses non-autoregressive models with connectionist temporal classification (ctc) @xcite , but these ctc-based models are designed for asr only. prior studies have also achieved promising results of ctc models for st only, but they mainly focus on specific language pairs at much smaller scales @xcite @xcite . some of them employ additional decoders @xcite or cross-attention layers @xcite , making the model more complicated. a natural question now arises: can we build a non-autoregressive encoder-only model for speechto-text generation in diverse languages and multiple tasks like whisper/owsm? this research problem has become increasingly important in the era of llms because large-scale pre-trained speech encoders can serve as an adapter between the speech and text modalities @xcite , providing a promising avenue towards general-purpose multi-modal foundation models @xcite . in this work, we propose owsm-ctc, a novel encoder-only speech foundation model based on multi-task self-conditioned ctc to imitate owsm's multilingual asr, any-to-any st, and lid functionalities. following previous encoderdecoder owsm v3.1 models @xcite , we train a 1b owsm-ctc model using 180k hours of public data covering 151 languages. extensive evaluations show that our owsm-ctc exhibits strong performance and efficiency. compared to the 1b owsm v3.1 medium model, owsm-ctc achieves comparable performance for asr and superior performance for various st directions (up to 24% relative improvement) while being more robust and showing 3 to 4 times inference speedup. owsm-ctc also improves the wer for longform asr and can be 20 times faster due to batched parallel decoding. owsm-ctc further outperforms the other baseline models on lid. our code, pre-trained model weights, and training logs will be publicly released to facilitate the development of large speech models. ## conclusion we propose owsm-ctc, a novel encoder-only speech foundation model built upon 180k hours of public audio data and open-source toolkits. owsm-ctc employs multi-task self-conditioned ctc for multilingual asr, any-to-any st, and lid. we conduct extensive experiments to compare owsm-ctc with the encoder-decoder owsm models trained on the same data. we find that owsm-ctc achieves competitive performance on asr and superior performance on st for both x-to-en (24% relative improvement) and en-to-x (13% relative improvement), while being more robust and 3 to 4 times faster at inference time. additionally, owsm-ctc improves the long-form asr wer with 20 times faster inference due to the batched parallel decoding. owsm-ctc also outperforms the baselines on lid. to promote open research on large speech models, we will publicly release our code, pre-trained model weights and training logs.