J oey S 2 T : Minimalistic Speech-to-Text Modeling with J oey NMT (2022)

Abstract: JoeyS2T is a JoeyNMT extension for speech-to-text tasks such as automatic speech recognition and end-to-end speech translation. It inherits the core philosophy of JoeyNMT, a minimalist NMT toolkit built on PyTorch, seeking simplicity and accessibility. JoeyS2T’s workflow is self-contained, starting from data pre-processing, over model training and prediction to evaluation, and is seamlessly integrated into JoeyNMT’s compact and simple code base. On top of JoeyNMT’s state-of-the-art Transformer-based Encoder-Decoder architecture, JoeyS2T provides speech-oriented components such as convolutional layers, SpecAugment, CTC-loss, and WER evaluation. Despite its simplicity compared to prior implementations, JoeyS2T performs competitively on English speech recognition and English-to-German speech translation benchmarks. The implementation is accompanied by a walk-through tutorial and available on https://github.com/may-/joeys2t .

## introduction end-to-end models recently have been shown to be able to outperform complex pipelines of individually trained components in many nlp tasks. for example, in the area of automatic speech recognition (asr) and speech translation (st), the performance gap between end-to-end models and cascaded pipelines, where an acoustic model is followed by an hmm for asr, or an asr model is followed by a machine translation (mt) model for st, seems to be closed @xcite . an end-to-end approach has several advantages over a pipeline approach: first, it mitigates error propagation through the pipeline. second, its data requirements are simpler since intermediate data interfaces to bridge components can be skipped. furthermore, intermediate components such as phoneme dictionaries in asr or transcriptions in st need significant amounts of ad-ditional human expertise to build. for end-to-end models, the overall model architecture is simpler, consisting of a unified end-to-end neural network. nonetheless, end-to-end components can be initialized from non end-to-end data, e.g., in audio encoding layers @xcite or text decoding layers @xcite . asr or st tasks usually have a higher entry barrier than mt, especially for novices who have little experience in machine learning, but also for nlp researchers who have previously only worked on text and not speech processing. this can also be seen in the population of the different tracks of nlp conferences. for example, the "speech and multimodality" track of acl 2022 had only a third of the number of papers in the "machine translation and multilinguality" track. 1 however, thanks to the end-to-end paradigm, those tasks are now more accessible for students or entry-level practitioners without huge resources, and without the experience of handling the different modules of a cascaded system or speech processing. the increased adoption of transformer architectures @xcite in both text @xcite and speech processing @xcite has further eased the transfer of knowledge between the two fields, in addition to making joint modeling easier and more unified. reviewing existing code bases for end-to-end asr and st-for example, deepspeech @xcite , espnet @xcite , fairseq s2t @xcite , neurst @xcite and speech-brain @xcite -it becomes apparent that the practical use of open-source toolkits still requires significant experience in navigating large-scale code, using complex data formats, preprocessing, neural text modeling, and speech processing in general. high code complexity and a lack of documentation are frustrating hurdles for novices. we propose joeys2t, a minimalist and accessible framework, to help novices get started with speech recognition and translation, to accelerate their learning process, and to make asr and st more accessible and transparent, that is directly targeting novices and their needs. we hope that making more accessible implementations will also have trickle-down effects of making the research built on top of it more accessible and more linguistically and geographically diverse @xcite . this effect has already been observed for the adoption of joeynmt for text mt for low-resource languages @xcite @xcite @xcite . furthermore, speech technology has an even higher potential for language inclusivity @xcite @xcite . ## design principles simplicity: we devoted considerable effort to keep joeys2t's module structure simple and flat. it directly employs the pytorch @xcite backend and has a low level of abstraction (details in section 4.6). joeys2t has a minimal list of external dependencies that can be easily installed via the pypi foot_1 tool. even for pre-processing, external dependencies on tools such as kaldi @xcite are avoided. for filterbank feature extraction, we use torchaudio foot_2 which is seamlessly integrated into pytorch. in contrast to other toolkits, speech modules extended in joeys2t are only built for speech-to-text modeling. it does not implement speech enhancement, nor speaker detection or speech generation. while this might appear like a limitation, we believe that the reduction of functionalities to a carefully identified minimum for st and asr is the key for increased accessibility. foot_3 accessibility: we also have written extensive documentation and walk-through tutorials to help newcomers become more familiar with speech technologies. joeys2t also provides pretrained models including configuration files which lower the barrier to get started. to guarantee the accessibility of the code, we open-sourced joeys2t under a very permissive license (apache 2.0). the joeys2t developer community actively supports user questions and requests. we maintain an open platform to discuss bug fixes, possible extensions etc. all contributions are first automatically controlled by the internal unit tests and will manually be reviewed by our team. reproducibility: to ensure that the reported results are comparable and reproducible, we release models trained on publicly available data. our evaluation metrics are described in detail (tokenization, punctuation handling etc.). all pre-and postprocessing scripts are published with a data download path and explicit hyperparameter configurations. we track all code changes in our repository and provide version information which is often a critical factor for reproducibility as bug fixes can affect evaluation scores. ## experimental results on benchmarks despite its simplicity, joeys2t achieves a performance on standard benchmarks that is comparable to other high-functional speech-to-text toolkits. ## conclusion & future work we described joeys2t, an extension of the joeynmt toolkit to the spoken language processing tasks asr and st. joeys2t is characterized by its minimalist design, prioritization of simplicity, accessibility and reproducibility in its code and documentation. the code is self-contained and requires minimal prior experience with speech or language processing. in benchmark evaluations, joeys2t performed comparable or superior to other asr or st code bases, while having much lower code complexity. while its functionality is kept minimal, support for state-of-the-art architectures such as wav2vec and conformer might be desired for future extensions.