The taste of IPA : Towards open-vocabulary keyword spotting and forced alignment in any language (2024)

Abstract: In this project, we demonstrate that phoneme-based models for speech processing can achieve strong crosslinguistic generalizability to unseen languages. We curated the IPAPACK, a massively multilingual speech corpora with phonemic transcriptions, encompassing more than 115 languages from diverse language families, selectively checked by linguists. Based on the IPAPACK, we propose CLAP-IPA, a multi-lingual phoneme-speech contrastive embedding model capable of open-vocabulary matching between arbitrary speech signals and phonemic sequences. The proposed model was tested on 95 unseen languages, showing strong generalizability across languages. Temporal alignments between phonemes and speech signals also emerged from contrastive training, enabling zeroshot forced alignment in unseen languages. We further introduced a neural forced aligner IPA-ALIGNER by finetuning CLAP-IPA with the Forward-Sum loss to learn better phone-to-audio alignment. Evaluation results suggest that IPA-ALIGNER can generalize to unseen languages without adaptation.

## introduction the diversity of human speech presents a formidable challenge to multilingual speech processing systems. recently, accumulating evidence indicates that scaling up the multilingual data can tremendously improve the performance of multilingual speech processing @xcite @xcite . however, it remains incredibly difficult, if not impossible, to gather large-scale data from every language in the world. it is becoming increasingly critical to develop speech processing systems that generalize to arbitrary unseen languages. despite the seeming diversity, sounds of human speech are highly constrained by the anatomical structure of the human vocal tract, which is universally shared by all humans @xcite . typological research has also shown that most, if not all, human speech can be represented by around 150 phonemes and diacritics @xcite . the limited degrees of freedom in human articulation have enabled phoneticians and linguists to craft universal symbolic representations of human speech, that is, the international phonetic alphabet (ipa) @xcite . prior studies have shown that phoneme-based asr models generalize to unseen languages @xcite @xcite . in this project, we aim to provide yet another positive answer to this central question: can we build multilingual speech processing systems that generalize to arbitrary languages through the use of universal ipa symbols? specifically, we focus on two classic tasks in speech processing, key word spotting (kws) and forced alignment. kws is a task of identifying specific keywords in streaming speech, whereas forced alignment refers to aligning intervals of a speech signal to a given sequence of phonetic symbols. both tasks are relevant in many practical applications such as voice assistant, speech synthesis, language documentation, etc. yet neither task has been tackled with general systems that generalize to all languages. this study represents an attempt to build crosslinguistically generalizable systems for kws and forced alignment. first, we present the ipapack, a multilingual speech corpora in 115 languages with phonemic transcriptions, totaling over 1000 hours and carefully checked by trained linguists. secondly, with the ipapack, we proposed contrastive language-audio pretraining with international phonetic alphabet (clap-ipa), a phonemeto-speech retrieval model with contrastive pretraining on phoneme-speech pairs. evaluations on 95 unseen languages suggest that clap-ipa is capable of performing zero-shot open-vocabulary kws in any language without adaption, including lan-guages not seen during training. thirdly, we also introduce a multilingual forced alignment model, ipa-aligner, that works for arbitrary languages. we noticed that alignments between phonemes and speech signals emerge from clap-ipa, even with only sequence-level contrastive learning. crosslinguistic zero-shot forced alignment can be achieved with clap-ipa. after finetuning clap-ipa with an alignment loss, we propose ipa-aligner that can provide crosslinguistic word-level and phone-level alignment generalizable to unseen languages. finally, our analysis indicates that phonemes, being shared across all languages, enhance knowledge transfer within training data, serving as more effective modeling units than texts in current multilingual tasks. we envision that our dataset and models will benefit more downstream tasks and applications in multilingual speech processing. to facilitate future research, we will release our dataset, scripts, and pre-trained models at: https://github.com/ lingjzhu/clap-ipa . ## dataset curation most speech corpora are distributed as audio-text pairs. in comparison, phonemically transcribed speech corpora are rare. unlike text transcription, transcribing speech signals into ipa, or phonemic transcriptions often require years of expertise in phonetics, making it hard to create highquality phonemic datasets at scale. however, these ipa symbols provide a universal representation of speech sounds such that any language can be transcribed symbolically. so ipa symbols can be used as a proxy to train multilingual speech processing systems. as a first step, we created large-scale phonemic transcriptions for public speech corpora, encompassing 115 languages across language families. the transcription can be automated through grapheme-to-phoneme conversion (g2p), a process of converting orthographic transcriptions into phonemic transcriptions through pronunciation dictionaries and/or statistical models. ## results in this section, we summarize the main results for kws and forced alignment. ## discussions in this section, we provide more in-depth answers to our research questions with the major findings of our experiments. can phoneme-based models generalize cross-linguistically? the evaluation results for clap-ipa and ipa-aligner in generally speaking, all clap-ipa models perform better on utterance-level datasets (fleurs-ipa and doreco-ipa) than on word-level datasets (mswc-ipa and ucla phonetic corpus), because the longer the phoneme sequence, the more likely that it is distinct in a pool of candidates. for utterance-level datasets, clap-ipa models achieve near-perfect scores on unseen languages (see for forced alignment, even the zero-shot predictions using clap-ipa can perform segmentation in unseen languages, especially at the word level. interestingly, there were no significant differences between performance over seen and unseen languages. though this result could be biased by the method seen-word seen-phone unseen-word unseen-phone ## conclusions with the carefully curated ipapack, we show that using ipa symbols as modeling units can effectively enable clap-ipa and ipa-aligner to generalize to unseen languages, highlighting the benefits of incorporating linguistic knowledge into deep learning methods. we believe that the ipapack has great potential to benefit more tasks in multilingual speech processing, such as multilingual phoneme recognition, speech synthesis, and documenting endangered languages. in the future, we will continue to expand our dataset and models to include more diverse languages. 9 ethical statement data governance we adhered strictly to ethical practices in curating our datasets. the original fleurs @xcite , mswc @xcite and voxcommunis @xcite corpora are distributed under the creative commons licenses. therefore, we are permitted to re-process and re-distribute the original dataset with proper attributions. some languages in the doreco corpus are under a creative commons non-commercial license. we reserved these languages to the test set in our corpora, such that our models have not been trained on data under commercially restrictive licenses. as required, we have also cited every individual language from the doreco corpus in potential impact we believe that our dataset and models will contribute to the endeavor of building fair and inclusive speech processing systems for all languages and facilitating the documentation of endangered languages. however, we are aware that multilingual keyword-spotting technology could potentially be misused as surveillance tools for monitoring speech recordings in more languages, posing risks to users.