textless-lib: a Library for Textless Spoken Language Processing (2022)

Abstract: Textless spoken language processing is an exciting area of research that promises to extend applicability of the standard NLP toolset onto spoken language and languages with few or no textual resources. Here, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in the area. We describe the building blocks that the library provides and demonstrate its usability by discuss three different use-case examples: (i) speaker probing, (ii) speech resynthesis and compression, and (iii) speech continuation. We believe that textless-lib substantially simplifies research the textless setting and will be handful not only for speech researchers but also for the NLP community at large.

## introduction textless spoken language modeling @xcite consists in jointly learning the acoustic and linguistic characteristics of a natural language from raw audio samples without access to textual supervision (e.g. lexicon or transcriptions). this area of research has been made possible by converging progress in self-supervised speech representation learning @xcite @xcite @xcite @xcite @xcite , language modeling @xcite @xcite @xcite , and speech synthesis @xcite @xcite @xcite . @xcite presented a generative spoken language modeling (gslm) pipeline trained from raw audio, consisting in a speech en-figure 1 : a visual description for textless modeling of spoken language. one can perform language modeling for speech continuations @xcite or a direct speech resynthesis @xcite . coder (converting speech to discrete units), a language model (based on units) and a decoder (converting units back to speech). these components enabled the generation of new speech by sampling units from the language model. @xcite proposed an improved encoder/decoder working from disentangled quantized content and f0 units and showed how such a system could be used for efficient audio compression. @xcite proposed a modified language model system capable of jointly modelling content units and f0 yielding expressive generations. lastly, @xcite demonstrated that the language model can be replaced by a sequence to sequence model achieving the first high quality speech emotion conversion system (including laughter and yawning). the textless approach has several advantages. first, it would be beneficial for the majority of the world's languages that do not have large textual resources or even a widely used standardized orthography (swiss german, dialectal arabic, igbo, etc.). despite being used by millions of people, these languages have little chance of being served by current 1 text-based technology. moreover, "high-resource" languages can benefit from such modeling where the oral and written forms are mismatched in terms of lexicon and syntax. second, directly modeling spoken language from raw audio allows us to go beyond lexical content and also model linguistically relevant signals such as prosodic features, intonation, non-verbal vocalizations (e.g., laughter, yawning, etc.), speaker identity, etc. all of these are virtually absent in text. although great progress has been made in modeling spoken language, it still requires domain expertise and involves a complicated setting. for instance, the official implementation of the gslm pipeline @xcite consists of roughly four different launching scripts with a few dozens of checkpoints. similarly, running the official implementation of @xcite , requires using four scripts from two different repositories. we present textless-lib, a pytorch library for textless spoken language processing. it makes processing, encoding, modeling, and generating of speech as simple as possible. with a few lines of code, one can perform speech continuation, audiobook compression, representation analysis by probing, speech-to-speech translation, etc. we provide all the necessary building blocks, example pipelines, and example tasks. we believe such a simple to use api will encourage both speech and nlp communities to deepen and extend the research work on modeling spoken language without text and unlock potential future research directions. ## background below we provide an overview of the common textless spoken language modeling pipeline. in a nutshell, such pipeline is usually comprised of: i) speech-to-units (s2u) encoders that automatically discover discrete representations or units which can be used to encode speech into "pseudo-text"; ii) units-to-units (u2u) models that are used for units modeling. this can take a form as unit-language-model (ulm) for speech continuation @xcite , sequenceto-sequence models for speech emotion conversion @xcite or translation tasks @xcite ; iii) units-to-speech (u2s) models to reconstruct back the speech signals. alternatively, one could drop the u2u component and perform a direct speech resynthesis @xcite . this can be used for speech compression, voice conversion, or develop- type model dataset encoders hubert ls-960 cpc ll-6k quantizers k-means ls-960 w. 50 units ls-960 w. 100 units ls-960 w. 200 units ls-960 w. 500 units f0 extract. yaapt -decoders tacotron2 lj speech waveglow lj speech ## library overview in this section, we present the textless-lib library, intending to simplify future research on textless spoken language modeling. additionally, the proposed package will remove the main barrier of processing and synthesizing speech, which requires domain expertise, for other language researchers (e.g., nlp researchers) who are interested in modeling spoken language, analyzing the learned representations, etc. to support the above, it is essential to provide the main building blocks described in section 2, together with pre-trained models, with minimal coupling between them (a list of the supported pretrained models can be seen on ## examples alongside the core functionality of the library, we provide a set of illustrative examples. the goal of these examples is two-fold: (a) to illustrate the usage of particular components of the library, and (b) to serve as a starter code for a particular type of application. for instance, a probing example (section 4.1) can be adapted for better studying used representations, while discrete resynthesis (section 4.2) could provide a starter code for an application operating on units (e.g., language modeling or a high-compression speech codec). ## discussion and future work we introduced textless-lib, a pytorch library aimed to advance research in textless modeling of spoken language, by simplifying textless processing and synthesizing spoken language. we described the main building blocks used to preprocess, quantize, and synthesize speech. to demonstrate the usability of the library, we provided three usage examples related to (i) representation probing, (ii) speech compression, and (iii) speech continuation. the proposed library greatly simplifies research in the textless spoken language processing, hence we believe it will be a handful not only for speech researchers but to the entire nlp community. as a future work for textless-lib we envision improving performance of the existing building blocks, adding new example tasks (e.g., translation @xcite or dialog @xcite ), extending the set of provided pre-trained models, and introducing the possibility of training the different components.