MUST & P - SRL : Multi-lingual and Unified Syllabification in Text and Phonetic Domains for Speech Representation Learning (2023)

Abstract: In this paper, we present a methodology for linguistic feature extraction, focusing particularly on automatically syllabifying words in multiple languages, with a design to be compatible with a forced-alignment tool, the Montreal Forced Aligner (MFA). In both the textual and phonetic domains, our method focuses on the extraction of phonetic transcriptions from text, stress marks, and a unified automatic syllabification (in text and phonetic domains). The system was built with open-source components and resources. Through an ablation study, we demonstrate the efficacy of our approach in automatically syllabifying words from several languages (English, French and Spanish). Additionally, we apply the technique to the transcriptions of the CMU ARCTIC dataset, generating valuable annotations available online (https://github.com/noetits/MUST_P-SRL) that are ideal for speech representation learning, speech unit discovery, and disentanglement of speech factors in several speech-related fields.

## introduction modern speech technologies have moved towards end-to-end models that constitutes black box systems that do not allow for explainability of the prediction or decisions. this lack of explainability started to raise a lot of concerns in the industry because of the need of identifying causes or reasons for decisions. this lead to the advent of the concept of explainable ai (xai) for which the goal is to discover ways to explains why a certain prediction was made by a system. for this, one avenue is the field of representation learning which incorporate unsupervised/selfsupervised learning, aiming to discover robust and meaningful representations for various tasks and analyze their relationship with expert knowledge (e.g. @xcite ). it is well known that in deep learning, learning knowledge can be tranferable from one task to other and self-supervised 1 https://github.com/noetits/must_p-srl learning is probably the most versatile transfer learning technique today. transfer learning @xcite is a widely used technique in deep learning for leveraging models trained on related tasks for which there exist abundant datasets towards tasks for which few labels exist. this principle has been applied successfully for speech technology application @xcite with few available data such as speech recognition for low resource languages, emotion recognition in speech @xcite , emotional or expressive speech synthesis @xcite or voice conversion @xcite . self-supervised learning is thus a specific form of transfer learning where a model is trained to learn representations of input data without the need for explicit supervision. these representations are the projection of the input data to a multidimensional space called latent space that captures information that is important for prediction of characteristics. there is however still a lot work to do to understand how these latent spaces are structured, what characteristics can be predicted, how can they be disentangled, etc. in this paper, we are particularly interested in providing a fine-grained expert annotations that can be aligned with a speech signal, allowing for exploration of relationships between speech representations and expert knowledge. to this end, our rich phonetic annotations, augmented with syllable and stress information, serve as strong supervisory signals. moreover, these phonetic transcriptions, tied to their written form, provide an explicit correspondence between the discrete symbols and their variably pronounced forms encountered in natural language. this could facilitate the discovery of speech units directly from the data. hence, this research can provide valuable insights and push the boundaries of current methods in automatic speech recognition, synthesis, and analysis. conducting linguistic feature extraction, such as phonetic transcriptions, syllable separations, and word stress, plays an essential role in a multitude of fields, such as speech representation learning, speech synthesis @xcite , speech recognition, and speaker identification. the ability to accurately mark syllable boundaries in words is fundamental for understanding language structure and its phonetic variations, which in turn aids in efficient decoding and analysis of speech data. among its potential use-cases, applications in the realms of second language learning and more specifically computer-assisted pronunciation training (capt) @xcite can greatly benefit from the reliable extraction, ensuring the development of effective learning materials that enhance pronunciation and overall language proficiency in learners. nevertheless, the extraction of linguistic features poses challenges due to the inherent complexity and variability observed in natural languages. dialectal variations, phonetic ambiguities, and inconsistencies in syllable boundaries are contributing factors that hinder the development of a reliable and consistent system for extracting linguistic features. moreover, there is a lack of resources that offer consistent phonetic transcriptions encompassing stress marks, phone boundaries, and syllable boundaries across both pronunciation and spelling domains. in this work, our goal is to define a methodology for linguistic feature extraction (phonetic transcriptions, stress marks, automatic syllabification in text and phonetic transcription domains) that is multilingual and compatible with forced-alignment tools. we have developed a process based on existing open-source building blocks that includes different steps and checks, as well as a consensus mechanism to extract the best possible linguistic features from text. the montreal forced aligner (mfa) @xcite is an essential tool in our analysis for its function in phonetic alignment, providing detailed pronunciation transcriptions. it is important to note that, while mfa is commonly used to align audio signals with corresponding text transcriptions, we consider that task to be already efficiently handled by mfa's acoustic models. our work aims to enrich this process: we focus on align- syllabification in text: -single vowel detection -look-up in syllabified words dataset -sonority extraction in text + language specific rules -ssp-dtw with sonorities in phonetics input sentence text normalization g2p: -pronunciation dictionary: mfa/cmu pronunciation dictionary -fallback on model stress mark estimation matching and consistency analysis output containing original and normalized text with segmented phonetics and text espeak ssp figure 1: block diagram of the linguistic feature extraction system described in section 3 ## related work automatic syllabification is a challenging task for natural language processing due to the ambiguity of syllable boundaries. different techniques have been developed to address this problem, including rule-based and data-driven approaches. in this section, we review some relevant studies on automatic syllabification in english, spanish, italian, and portuguese. for english, the study presented in @xcite compares five different algorithms, including two rule-based approaches and three datadriven techniques. the study finds that data-driven methods outperform rule-based systems in terms of word and juncture accuracy. furthermore, syllabification in the pronunciation domain is easier than in the spelling domain. the study also highlights the challenge of establishing a gold standard corpus for evaluation due to the lack of consensus in the entries of multiple lexical databases. however, in their experiment, they apply the two rulebased algorithms in the spelling domain without any adaptation, and they do not consider the use of the sonority sequencing principle. the sonority sequencing principle (ssp) @xcite ) is a widely used rule for syllabification, which states that syllables are formed by increasing then decreasing sonority. it is based on the sonority hierarchy, which assigns a relative sonority value to each phone. vowels have the highest sonority, followed by approximants (such as /r/ and /w/), fricatives, nasals, and finally stops, which have the lowest sonority. the linguistic literature identified exceptions to this principle, the main one being probably the sibilant-stop consonant cluster @xcite @xcite . implementations of the principle with processing of these exceptions been successfully applied for automatic syllabification in several languages in the pronunciation domain with very high word accuracies @xcite @xcite . but it has also been applied in the spelling domain with some success for some languages. in spanish, @xcite points out that syllabification follows basic rules but may deviate due to various factors, such as diphthongs or hiatuses. some variations in syllabification are also related to geographical and dialectal criteria. therefore, automatic syllabification in spanish requires taking into account these variations. for italian, @xcite presents a rule-based method that uses the sonority sequencing principle (ssp) and additional rules specific to italian. the study evaluates their method on a dataset of sentences that were manually syllabified and reports an accuracy of 0.98-1 for some of the subjects. we could not find an application of ssp in the spelling domain in english. the reason is maybe because a naive application of ssp in the spelling domain would not perform very well. many data-driven syllabification methods using different levels of complexities of machine/deep learning models, that have the potential to be applied to several languages, have been developed but mainly for the phonetic domain only @xcite @xcite . in this literature review, we did not find any method that is capable of syllabification in both pronunciation and spelling domains and study the consistency between them. in this work, we thus propose a methodology for a unified automatic syl-labification and experiment it in several languages. ## system the proposed methodology for linguistic feature extraction is illustrated in figure 1 . it includes several steps: text normalization, grapheme-tophoneme (g2p) conversion, syllabification in the phonetic domain, and syllabification in the text domain. lastly, a consistency analysis is conducted to identify words with inconsistent syllable counts, facilitating manual correction of the remaining exceptional cases. the system is designed to be multilingual and compatible with forced-alignment tools, namely montreal forced aligner (mfa). ## experiments to evaluate the quality of an automatic syllabification algorithm, two measures are typically used: word accuracy and juncture accuracy. word accuracy measures the proportion of words for which the number of syllables is exactly the same as a gold standard. juncture accuracy measures the proportion of junctures that are the same as a gold standard. in this study, we propose to measure word accuracy between the syllabified text of our methodology and the result of the application of the sonority sequencing principle in the pronunciation domain. this is backed by the literature, as the number of syllables extracted in the phonetic domain is highly reliable. this measure allows for reproducibility and avoids comparison with a gold standard annotated by humans, which is also imperfect and inconsistent. our consensus mechanism allowed us to detect errors that can complement syllabified text corpora or start corpora of edge cases for new languages. ## conclusions this study introduced a novel, multilingual methodology for linguistic feature extraction, designed to be compatible with forced-alignment tools. our approach effectively extracted essential linguistic features, including phonetic transcriptions, stress marks, and automatic syllabification in both text and phonetic domains. the methodology integrated various techniques, such as text normalization, grapheme-to-phoneme conversion, syllabification in the phonetic and text domains, and a consensus analysis to identify inconsistencies. our ablation study demonstrated the efficacy of the proposed methodology in automatically syllabifying words across multiple languages. the optimal performance was achieved by combining corpus lookup and dynamic time warping (dtw) on sonority sequences. this approach can be further enhanced by progressively incorporating edge cases into the training dataset. by applying our methodology to the cmu arc-tic dataset, we generated valuable data that can benefit various speech-related research domains, available online foot_8 . our unified phonetization and syllabification annotations have the potential to advance speech representation learning and disentangle different factors in speech technologies, such as speech synthesis and speech analysis tasks.