Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in A rabic (2024)

Abstract: This paper presents a novel Dialectal Sound and Vowelization Recovery framework, designed to recognize borrowed and dialectal sounds within phonologically diverse and dialect-rich languages, that extends beyond its standard orthographic sound sets. The proposed framework utilized quantized sequence of input with(out) continuous pretrained self-supervised representation. We show the efficacy of the pipeline using limited data for Arabic, a dialect-rich language containing more than 22 major dialects. Phonetically correct transcribed speech resources for dialectal Arabic is scare. Therefore, we introduce ArabVoice15, a first of its kind, curated test set featuring 5 hours of dialectal speech across 15 Arab countries, with phonetically accurate transcriptions, including borrowed and dialect-specific sounds. We described in detail the annotation guideline along with the analysis of the dialectal confusion pairs. Our extensive evaluation includes both subjective – human perception tests and objective measures. Our empirical results, reported with three test sets, show that with only one and half hours of training data, our model improve character error rate by ≈7% in ArabVoice15 compared to the baseline.

## introduction self-supervised learning (ssl) paradigm has transformed speech research and technology, achieving remarkable performance @xcite while reducing the dependency on extensively annotated datasets @xcite . the ssl models excel at discerning the underlying acoustic properties in both frames and utterance level @xcite @xcite irrespective of language. phonetic information is sailent and preserved even when these continuous representations are mapped to a finite set of codes via vector quantization @xcite @xcite . this allows the learning paradigm to leverage unlabeled data to discover units that capture meaningful phonetic contrasts. leveraging insights from acoustic unit discovery @xcite @xcite @xcite , unsupervised speech recognition @xcite @xcite @xcite , and phoneme segmentation @xcite @xcite have utilized quantized discrete units for various purposes. these include (i) pretraining the ssl model @xcite , (ii) employing acoustic unit discovery as a training objective @xcite , and (iii) utilizing discrete labels for training phoneme recognition and automatic speech recognition @xcite @xcite . inspired by previous research, we employ ssl representations and vector quantization to recognize acoustic units in phonologically diverse spoken dialects, extending beyond their standard orthographic sound sets. we introduce a simple yet potent network leveraging ssl and a discrete codebook to recognize these non-orthographic dialectal and borrowed sounds with minimal labeled data. arabic is an appropriate language choice for the task. the language has a rich tapestry of dialects, each with its unique characteristics in phonology, morphology, syntax, and lexicon @xcite . these dialects 2 differ not only among themselves but also when compared to modern standard arabic (msa). while msa prevails in official and educational domains, dialectal arabic (da) serves as the means for daily communication. the diversity in pronunciation and phoneme sets for da goes beyond standardized msa sound sets. moreover, to add to the challenges, da follows no standard orthography. therefore, despite the abundance of da speech data in online platforms, accurately (phonetically correct) transcribed resources are scarce, categorizing da among the low-resource languages. to bridge this gap, we introduce the arabic "dialectal sound and vowelization recovery" (dsvr) framework. the proposed framework exploits the frame-level ssl embeddings and quantizes them to create a handful of discrete labels using k-means model. these discrete labels are then fed (can be in combination with ssl embeddings) as input to a transformer-based dialectal unit and vowel recognition (dvr) model. we show its efficacy for (a) dialectal and borrowed sound recovery; and (b) vowelization restoration capabilities with only 1 hour 30 minutes of training data. we introduced arabic dialectal test set -"arabvoice15", a collection of 5 hours of dialectal speech and verbatim transcription with recovered dialectal and borrowed sounds from 15 arab countries. for vowelization restoration, we tested on 1 hour of speech data, sampled from commonvoice-ar @xcite , transcribed by restoring short vowels. our paper describes the phonetic rules adopted, special sounds considered along with detailed annotation guidelines for designing these test sets. furthermore, we evaluate the quality of the intermediate discrete labels using human perceptual evaluation, in addition to other purity and clustering-based measures. we observed that these discrete labels can capture speaker-invariant, distinct acoustic, and linguistic information while preserving the temporal information. consequently, encapsulating the discriminate acoustic unit properties, which can be used to recover dialectal missing sounds. our empirical results suggest that dsvr can exploit unlabeled data to design the codebook and then with a small amount of annotated data, a unit recognizer can be trained. ## arabic sounds the exploration of phonotactic variations across arabic dialects, including msa and other regional dialects offers a rich field of study within the domain of arabic linguistics. these variations are not merely lexical, but phonetic and in many cases deeply embedded in the phonological rules that dictate the permissible combinations and sequences of sounds within each dialect @xcite . ## arabvoice15 dataset spoken da remains a low-resource language primarily due to the scarcity of transcription that can faithfully capture the diverse regional and borrowed sounds in the standard written format. such lack of data posses significant challenge for speech and linguistic research and evaluation. in this study, we address this challenge by designing and developing arabvoice15 test set. furthermore, we have also enhanced a subset of the existing arabic commonvoice @xcite , ar:cv r dataset with restored vowels, borrowed and dialectal sounds. in the following sections, we will discuss the datasets, preprocessing steps along with in detail annotation guidelines. arabvoice15 is a collection of 5 hours of speech utterances randomly selected from testset of adi17 figure 2: baseline and dvr -discrete and joint model (ali et al., 2019) dataset, widely used for dialect identification task. for the arabvoice15, we selected a total of 2500 utterance, ≈ 146(±3.6) utterance from each of the 15 arab countries including: algeria (alg), egypt (egy), iraq (ira), jordan (jor), saudi arabia (ksa), kuwait (kuw), lebanon (leb), libya (lib), morocco (mor), palestine (pal), qatar (qat), sudan (sud), syria (syr), united arab emirates (uae), and yemen (yem). the average utterance duration: 7-8 seconds. as for ar : cv r , we randomly extracted 21.38 hours from the ar:cv trainset, which we then mannually annotated at both verbatim and vowelized level (test ≈ 1hr). data verbatim pre-processing we present a set of rules employed for data normalization, aiming to reduce annotators' tasks through a rule-based phonemic letter-to-sound approach in arabic, as detailed in @xcite . for vowelization, we initially applied diacritization (aka vowelization or vowel restoration) module present in the farasa tool @xcite . we then applied the following rule-based phonemic letterto-sound function to our dataset. this step also removed any arabic letters that are not traditionally pronounced in spoken conversation. ## results and discussion number of discrete codes in codebook we reported the dbindex for the codebook sizes @xmath0, 256, 512} in perceptual test of codebook we averaged annotator judgments across four categories for all mix clusters, revealing no clear majority and highlighting the listeners' difficulty in categorically labeling audio within these clusters. in aligned with @xcite ; @xcite , we also conclude that these mixed labels genuinely exist and cannot be precisely characterized by any conventional given label. we present some of our findings of the perceptual test in figure 3 for 5 different mix clusters with average judgment per category. impact of training data size performance for short vowel restoration for short vowel restoration (in ## conclusion in this study, we propose a novel dialectal sound and short vowel recovery framework that utilizes a handful of discrete codes to represent the variability in dialectal arabic. we also observed with only 256 discrete labels, the borrowed and dialectal sound recognition model outperforms both baseline and joint (discrete code with frame-level ssl representation) models by ≈ 7% cer improvement. for restoring vowels, we noticed ssl embeddings play a bigger role. our findings indicate the efficacy of the discrete model with small training datasets. to foster further research in dialectal arabic, we introduced, benchmarked, and released arabvoice15 -a dialectal verbatim transcription dataset containing utterances from 15 arab countries. in the future, we will apply the framework to more dialects and other dialectal languages.